2023-03-03T06:07:00.446 INFO:root:teuthology version: 0.0.1.dev49+gc7c3cc8 2023-03-03T06:07:00.447 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi --verbose --archive /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140 --owner scheduled_vshankar@teuthology --description fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} -- /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140/orig.config.yaml 2023-03-03T06:07:00.554 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-03T06:07:00.633 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140 branch: wip-vshankar-testing-20230303.023823 description: fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} email: vshankar@redhat.com first_in_suite: false job_id: '7192140' kernel: client: sha1: distro flavor: default kdb: true sha1: 75534f74ac3dd06f86697ff862e8be2945b3c8e6 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-vshankar-testing-20230303.023823 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20230303.023823 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20230303.023823 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6 timestamp: 2023-03-03_04:39:14 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.31660 2023-03-03T06:07:00.633 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa; will attempt to use it 2023-03-03T06:07:00.635 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa/tasks 2023-03-03T06:07:00.636 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-03T06:07:00.638 INFO:teuthology.task.internal:Checking packages... 2023-03-03T06:07:00.670 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '472aaef4ec3b3a2c29f73ca67854785a910f1f9b' 2023-03-03T06:07:00.670 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-03T06:07:00.671 INFO:teuthology.packaging:ref: None 2023-03-03T06:07:00.671 INFO:teuthology.packaging:tag: None 2023-03-03T06:07:00.671 INFO:teuthology.packaging:branch: wip-vshankar-testing-20230303.023823 2023-03-03T06:07:00.672 INFO:teuthology.packaging:sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T06:07:00.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20230303.023823 2023-03-03T06:07:00.821 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-2739.g472aaef4 2023-03-03T06:07:00.821 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-03T06:07:00.828 INFO:teuthology.task.internal:no buildpackages task found 2023-03-03T06:07:00.829 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-03T06:07:00.883 INFO:teuthology.task.internal:Saving configuration 2023-03-03T06:07:00.903 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-03T06:07:00.942 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-03T06:07:00.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 06:01:10.578659', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-03T06:07:00.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 06:01:10.580242', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-03T06:07:00.999 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-03T06:07:01.013 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-03-03T06:07:01.014 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-03-03T06:07:01.014 INFO:teuthology.run_tasks:Running task console_log... 2023-03-03T06:07:01.207 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa9142674c0>, signals=[15]) 2023-03-03T06:07:01.209 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-03T06:07:01.220 INFO:teuthology.task.internal:Opening connections... 2023-03-03T06:07:01.220 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2023-03-03T06:07:01.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T06:07:01.295 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2023-03-03T06:07:01.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T06:07:01.365 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-03T06:07:01.382 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2023-03-03T06:07:01.402 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2023-03-03T06:07:01.403 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2023-03-03T06:07:01.465 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Red Hat Enterprise Linux" 2023-03-03T06:07:01.465 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8.6 (Ootpa)" 2023-03-03T06:07:01.466 INFO:teuthology.orchestra.run.smithi090.stdout:ID="rhel" 2023-03-03T06:07:01.466 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="fedora" 2023-03-03T06:07:01.466 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8.6" 2023-03-03T06:07:01.466 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2023-03-03T06:07:01.466 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-03T06:07:01.466 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2023-03-03T06:07:01.467 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-03T06:07:01.467 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.redhat.com/" 2023-03-03T06:07:01.467 INFO:teuthology.orchestra.run.smithi090.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-03T06:07:01.467 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T06:07:01.467 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:07:01.467 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T06:07:01.467 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-03T06:07:01.468 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-03T06:07:01.468 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-03T06:07:01.468 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2023-03-03T06:07:01.497 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-03-03T06:07:01.517 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-03-03T06:07:01.518 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-03-03T06:07:01.577 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux" 2023-03-03T06:07:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.6 (Ootpa)" 2023-03-03T06:07:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel" 2023-03-03T06:07:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora" 2023-03-03T06:07:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.6" 2023-03-03T06:07:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2023-03-03T06:07:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-03T06:07:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/" 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-03T06:07:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-03T06:07:01.580 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-03T06:07:01.580 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-03-03T06:07:01.604 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-03T06:07:01.620 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-03T06:07:01.679 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-03T06:07:01.679 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2023-03-03T06:07:01.682 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2023-03-03T06:07:01.701 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-03T06:07:01.741 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-03T06:07:01.741 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2023-03-03T06:07:01.745 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2023-03-03T06:07:01.775 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-03T06:07:01.865 INFO:teuthology.run_tasks:Running task kernel... 2023-03-03T06:07:01.893 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'} 2023-03-03T06:07:01.894 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-03-03T06:07:01.894 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-03-03T06:07:01.894 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T06:07:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T06:07:01.913 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2023-03-03T06:07:01.980 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-03T06:07:01.981 INFO:teuthology.task.kernel:Running kernel on smithi090: 4.18.0-372.9.1.el8.x86_64 2023-03-03T06:07:01.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2023-03-03T06:07:03.385 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2023-03-03T06:07:03.386 INFO:teuthology.orchestra.run.smithi090.stdout:Unable to read consumer identity 2023-03-03T06:07:07.186 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 1.7 kB/s | 3.5 kB 00:02 2023-03-03T06:07:08.244 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 13 MB/s | 14 MB 00:01 2023-03-03T06:07:11.370 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 442 kB/s | 24 kB 00:00 2023-03-03T06:07:12.072 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-03T06:07:12.127 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-03-03T06:07:12.128 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2023-03-03T06:07:12.128 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-03-03T06:07:12.237 DEBUG:teuthology.orchestra.run.smithi090:> echo no | sudo yum reinstall kernel || true 2023-03-03T06:07:12.566 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2023-03-03T06:07:12.567 INFO:teuthology.orchestra.run.smithi090.stdout:Unable to read consumer identity 2023-03-03T06:07:12.817 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:01 ago on Fri 03 Mar 2023 06:07:11 AM UTC. 2023-03-03T06:07:12.859 INFO:teuthology.orchestra.run.smithi090.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-03T06:07:12.924 INFO:teuthology.orchestra.run.smithi090.stderr:Error: No packages marked for reinstall. 2023-03-03T06:07:12.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum reinstall -y kernel || true 2023-03-03T06:07:13.299 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2023-03-03T06:07:13.300 INFO:teuthology.orchestra.run.smithi090.stdout:Unable to read consumer identity 2023-03-03T06:07:13.547 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:02 ago on Fri 03 Mar 2023 06:07:11 AM UTC. 2023-03-03T06:07:13.586 INFO:teuthology.orchestra.run.smithi090.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-03T06:07:13.612 INFO:teuthology.orchestra.run.smithi090.stderr:Error: No packages marked for reinstall. 2023-03-03T06:07:13.658 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel --last | head -n 1 2023-03-03T06:07:13.739 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-03T06:07:13.740 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-03T06:07:13.741 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-03T06:07:13.741 DEBUG:teuthology.parallel:result is None 2023-03-03T06:07:13.741 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-03T06:07:13.778 INFO:teuthology.task.internal:Creating test directory... 2023-03-03T06:07:13.779 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T06:07:13.782 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T06:07:13.802 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-03T06:07:13.855 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-03T06:07:13.943 INFO:teuthology.task.internal:Creating archive directory... 2023-03-03T06:07:13.943 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T06:07:13.946 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T06:07:13.989 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-03T06:07:14.037 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-03T06:07:14.038 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T06:07:14.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T06:07:14.062 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T06:07:14.065 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T06:07:14.099 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T06:07:14.110 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T06:07:14.147 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T06:07:14.166 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T06:07:14.167 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-03T06:07:14.190 INFO:teuthology.task.internal:Configuring sudo... 2023-03-03T06:07:14.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T06:07:14.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T06:07:14.221 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-03T06:07:14.235 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-03T06:07:14.235 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T06:07:14.260 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T06:07:14.279 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T06:07:14.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T06:07:14.417 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T06:07:14.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T06:07:14.542 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:07:14.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T06:07:14.609 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T06:07:14.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T06:07:14.717 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T06:07:14.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T06:07:14.847 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:07:14.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T06:07:14.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2023-03-03T06:07:14.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2023-03-03T06:07:14.985 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T06:07:15.058 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T06:07:15.406 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-03T06:07:15.446 INFO:teuthology.task.internal:Starting timer... 2023-03-03T06:07:15.447 INFO:teuthology.run_tasks:Running task pcp... 2023-03-03T06:07:15.620 INFO:teuthology.run_tasks:Running task selinux... 2023-03-03T06:07:15.793 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-03-03T06:07:15.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2023-03-03T06:07:15.851 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2023-03-03T06:07:15.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2023-03-03T06:07:15.937 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2023-03-03T06:07:15.938 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-03T06:07:15.939 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2023-03-03T06:07:15.965 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2023-03-03T06:07:15.966 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2023-03-03T06:07:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2023-03-03T06:07:15.992 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2023-03-03T06:07:15.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T06:07:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T06:07:16.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T06:07:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T06:07:16.060 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-03T06:07:16.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2023-03-03T06:07:16.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2023-03-03T06:07:16.130 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-03T06:07:16.171 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-03-03T06:07:16.172 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-03T06:07:16.196 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-03T06:07:16.198 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-03T06:14:38.039 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2023-03-03T06:14:38.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-03-03T06:14:38.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T06:14:38.111 DEBUG:teuthology.orchestra.run.smithi090:> true 2023-03-03T06:14:38.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-03-03T06:14:38.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T06:14:38.247 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-03-03T06:14:38.319 INFO:teuthology.run_tasks:Running task clock... 2023-03-03T06:14:38.363 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-03T06:14:38.364 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T06:14:38.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T06:14:38.367 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T06:14:38.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T06:14:38.397 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T06:14:38.400 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T06:14:38.413 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T06:14:38.414 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T06:14:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2023-03-03T06:14:38.484 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2023-03-03T06:14:38.495 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2023-03-03T06:14:38.496 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2023-03-03T06:14:38.509 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T06:14:38.511 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T06:14:38.523 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T06:14:38.524 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T06:14:38.593 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2023-03-03T06:14:38.595 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-03-03T06:14:38.596 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T06:14:38.596 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2023-03-03T06:14:38.596 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.597 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.597 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.597 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.598 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T06:14:38.598 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-03-03T06:14:38.598 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.599 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.599 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.599 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T06:14:38.600 INFO:teuthology.run_tasks:Running task install... 2023-03-03T06:14:38.642 DEBUG:teuthology.task.install:project ceph 2023-03-03T06:14:38.642 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}} 2023-03-03T06:14:38.643 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'} 2023-03-03T06:14:38.643 INFO:teuthology.task.install:Using flavor: default 2023-03-03T06:14:38.649 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-03T06:14:38.649 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-03-03T06:14:38.650 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T06:14:38.651 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T06:14:38.653 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T06:14:38.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T06:14:38.791 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T06:14:38.792 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T06:14:38.800 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T06:14:38.800 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T06:14:38.986 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T06:14:38.986 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:14:38.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T06:14:38.990 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T06:14:38.990 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:14:38.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T06:14:39.018 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T06:14:39.018 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T06:14:39.019 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T06:14:39.020 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T06:14:39.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T06:14:39.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T06:14:39.163 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T06:14:39.165 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T06:14:39.229 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2023-03-03T06:14:39.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2023-03-03T06:14:39.231 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2023-03-03T06:14:39.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2023-03-03T06:14:39.601 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-03-03T06:14:39.601 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2023-03-03T06:14:39.860 INFO:teuthology.orchestra.run.smithi090.stdout:44 files removed 2023-03-03T06:14:39.869 INFO:teuthology.orchestra.run.smithi121.stdout:44 files removed 2023-03-03T06:14:39.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T06:14:39.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T06:14:40.233 INFO:teuthology.orchestra.run.smithi090.stdout:Updating Subscription Management repositories. 2023-03-03T06:14:40.243 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-03-03T06:14:41.284 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 113 kB/s | 80 kB 00:00 2023-03-03T06:14:41.381 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 102 kB/s | 80 kB 00:00 2023-03-03T06:14:41.793 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 20 kB/s | 9.4 kB 00:00 2023-03-03T06:14:41.915 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 19 kB/s | 9.4 kB 00:00 2023-03-03T06:14:42.419 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2023-03-03T06:14:42.507 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2023-03-03T06:14:44.753 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-03-03T06:14:44.858 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-03-03T06:14:46.247 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 14 MB 00:01 2023-03-03T06:14:46.450 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 14 MB 00:01 2023-03-03T06:14:49.339 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 286 kB/s | 24 kB 00:00 2023-03-03T06:14:49.532 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 244 kB/s | 24 kB 00:00 2023-03-03T06:14:50.409 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 MB/s | 53 MB 00:00 2023-03-03T06:14:50.416 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 53 MB 00:01 2023-03-03T06:14:58.426 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 47 MB 00:00 2023-03-03T06:14:58.560 INFO:teuthology.orchestra.run.smithi090.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 54 MB/s | 47 MB 00:00 2023-03-03T06:15:09.182 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T06:15:09.182 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T06:15:09.197 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-03T06:15:09.198 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-03T06:15:09.198 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-03T06:15:09.199 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T06:15:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T06:15:09.205 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T06:15:09.206 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T06:15:09.339 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-03-03T06:15:09.352 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-03-03T06:15:09.352 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-03-03T06:15:09.352 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-03-03T06:15:09.352 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-03-03T06:15:09.352 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-03T06:15:09.353 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-03T06:15:09.353 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T06:15:09.353 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T06:15:09.353 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T06:15:09.353 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T06:15:09.353 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T06:15:09.353 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T06:15:09.354 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T06:15:09.354 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T06:15:09.354 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T06:15:09.354 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T06:15:09.354 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T06:15:09.354 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T06:15:09.354 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T06:15:09.355 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T06:15:09.355 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T06:15:09.355 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-03T06:15:09.355 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-03T06:15:09.355 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-03T06:15:09.355 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T06:15:09.355 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-03T06:15:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T06:15:09.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T06:15:09.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T06:15:09.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T06:15:09.357 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T06:15:09.357 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T06:15:09.358 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T06:15:09.358 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-03T06:15:09.358 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2023-03-03T06:15:09.359 INFO:teuthology.orchestra.run.smithi121.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-03T06:15:09.359 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T06:15:09.359 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T06:15:09.359 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-03-03T06:15:09.359 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-03T06:15:09.359 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T06:15:09.359 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T06:15:09.360 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T06:15:09.360 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T06:15:09.360 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T06:15:09.360 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T06:15:09.361 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T06:15:09.361 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T06:15:09.361 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-03T06:15:09.361 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T06:15:09.361 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-03T06:15:09.361 INFO:teuthology.orchestra.run.smithi121.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-03T06:15:09.361 INFO:teuthology.orchestra.run.smithi121.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-03T06:15:09.362 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T06:15:09.362 INFO:teuthology.orchestra.run.smithi121.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-03T06:15:09.362 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-03T06:15:09.362 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T06:15:09.362 INFO:teuthology.orchestra.run.smithi121.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-03T06:15:09.362 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-03T06:15:09.362 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-03T06:15:09.363 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T06:15:09.363 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-03T06:15:09.363 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T06:15:09.363 INFO:teuthology.orchestra.run.smithi121.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-03T06:15:09.363 INFO:teuthology.orchestra.run.smithi121.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-03T06:15:09.363 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T06:15:09.363 INFO:teuthology.orchestra.run.smithi121.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-03T06:15:09.364 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T06:15:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-03T06:15:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T06:15:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T06:15:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T06:15:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T06:15:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T06:15:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T06:15:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-03T06:15:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T06:15:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T06:15:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-03T06:15:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T06:15:09.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T06:15:09.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T06:15:09.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-03T06:15:09.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-03T06:15:09.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-03T06:15:09.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T06:15:09.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T06:15:09.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T06:15:09.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-03T06:15:09.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-03T06:15:09.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T06:15:09.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T06:15:09.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-03T06:15:09.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-03T06:15:09.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-03T06:15:09.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T06:15:09.369 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T06:15:09.370 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-03T06:15:09.370 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-03T06:15:09.370 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-03T06:15:09.370 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-03T06:15:09.371 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-03T06:15:09.371 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-03T06:15:09.371 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T06:15:09.372 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T06:15:09.372 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T06:15:09.372 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T06:15:09.372 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-03T06:15:09.373 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T06:15:09.373 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T06:15:09.373 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T06:15:09.373 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T06:15:09.374 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T06:15:09.374 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T06:15:09.374 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T06:15:09.375 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T06:15:09.375 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-03T06:15:09.375 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T06:15:09.375 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-03T06:15:09.376 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T06:15:09.376 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-03T06:15:09.376 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-03T06:15:09.376 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-03T06:15:09.377 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2023-03-03T06:15:09.377 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql 10 2023-03-03T06:15:09.377 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:15:09.377 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-03-03T06:15:09.378 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-03-03T06:15:09.378 INFO:teuthology.orchestra.run.smithi121.stdout:Install 128 Packages 2023-03-03T06:15:09.378 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 3 Packages 2023-03-03T06:15:09.379 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:15:09.379 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 203 M 2023-03-03T06:15:09.380 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-03-03T06:15:09.493 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T06:15:09.494 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T06:15:09.508 INFO:teuthology.orchestra.run.smithi090.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-03T06:15:09.510 INFO:teuthology.orchestra.run.smithi090.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-03T06:15:09.511 INFO:teuthology.orchestra.run.smithi090.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-03T06:15:09.512 INFO:teuthology.orchestra.run.smithi090.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T06:15:09.514 INFO:teuthology.orchestra.run.smithi090.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T06:15:09.518 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T06:15:09.519 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T06:15:09.654 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2023-03-03T06:15:09.666 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================================================== 2023-03-03T06:15:09.666 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2023-03-03T06:15:09.667 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================================================== 2023-03-03T06:15:09.667 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2023-03-03T06:15:09.667 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-03T06:15:09.667 INFO:teuthology.orchestra.run.smithi090.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-03T06:15:09.667 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T06:15:09.667 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T06:15:09.667 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T06:15:09.668 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T06:15:09.668 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T06:15:09.668 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T06:15:09.668 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T06:15:09.668 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T06:15:09.668 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T06:15:09.668 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T06:15:09.669 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T06:15:09.669 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T06:15:09.669 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T06:15:09.669 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T06:15:09.670 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T06:15:09.671 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-03T06:15:09.671 INFO:teuthology.orchestra.run.smithi090.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-03T06:15:09.671 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-03T06:15:09.671 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T06:15:09.671 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T06:15:09.671 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T06:15:09.672 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T06:15:09.674 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T06:15:09.674 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T06:15:09.674 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T06:15:09.674 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T06:15:09.675 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T06:15:09.675 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-03T06:15:09.675 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2023-03-03T06:15:09.675 INFO:teuthology.orchestra.run.smithi090.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-03T06:15:09.676 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T06:15:09.676 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T06:15:09.676 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2023-03-03T06:15:09.676 INFO:teuthology.orchestra.run.smithi090.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-03T06:15:09.677 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T06:15:09.678 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T06:15:09.678 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T06:15:09.678 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T06:15:09.678 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T06:15:09.678 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T06:15:09.678 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T06:15:09.679 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T06:15:09.679 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-03T06:15:09.679 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T06:15:09.679 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-03T06:15:09.679 INFO:teuthology.orchestra.run.smithi090.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-03T06:15:09.680 INFO:teuthology.orchestra.run.smithi090.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-03T06:15:09.680 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T06:15:09.680 INFO:teuthology.orchestra.run.smithi090.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-03T06:15:09.680 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-03T06:15:09.681 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T06:15:09.681 INFO:teuthology.orchestra.run.smithi090.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-03T06:15:09.681 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-03T06:15:09.681 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-03T06:15:09.681 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T06:15:09.681 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-03T06:15:09.682 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T06:15:09.682 INFO:teuthology.orchestra.run.smithi090.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-03T06:15:09.682 INFO:teuthology.orchestra.run.smithi090.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-03T06:15:09.682 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T06:15:09.683 INFO:teuthology.orchestra.run.smithi090.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-03T06:15:09.683 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-03T06:15:09.683 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-03T06:15:09.683 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-03T06:15:09.684 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-03T06:15:09.685 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-03T06:15:09.685 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-03T06:15:09.685 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-03T06:15:09.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T06:15:09.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-03T06:15:09.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T06:15:09.685 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T06:15:09.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T06:15:09.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T06:15:09.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T06:15:09.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T06:15:09.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-03T06:15:09.686 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T06:15:09.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T06:15:09.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-03T06:15:09.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T06:15:09.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T06:15:09.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T06:15:09.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-03T06:15:09.687 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-03T06:15:09.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-03T06:15:09.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T06:15:09.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T06:15:09.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T06:15:09.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-03T06:15:09.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-03T06:15:09.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T06:15:09.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T06:15:09.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-03T06:15:09.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-03T06:15:09.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-03T06:15:09.690 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T06:15:09.690 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T06:15:09.690 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-03T06:15:09.690 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-03T06:15:09.690 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-03T06:15:09.690 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-03T06:15:09.690 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T06:15:09.691 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T06:15:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T06:15:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T06:15:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T06:15:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T06:15:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T06:15:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T06:15:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-03T06:15:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T06:15:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-03T06:15:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T06:15:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-03T06:15:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-03T06:15:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-03T06:15:09.693 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling module streams: 2023-03-03T06:15:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql 10 2023-03-03T06:15:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:15:09.694 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2023-03-03T06:15:09.694 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================================================== 2023-03-03T06:15:09.694 INFO:teuthology.orchestra.run.smithi090.stdout:Install 128 Packages 2023-03-03T06:15:09.694 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 3 Packages 2023-03-03T06:15:09.695 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:15:09.696 INFO:teuthology.orchestra.run.smithi121.stdout:(1/131): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 21 kB/s | 6.4 kB 00:00 2023-03-03T06:15:09.696 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 203 M 2023-03-03T06:15:09.696 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2023-03-03T06:15:10.028 INFO:teuthology.orchestra.run.smithi090.stdout:(1/131): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 18 kB/s | 6.4 kB 00:00 2023-03-03T06:15:10.537 INFO:teuthology.orchestra.run.smithi090.stdout:(2/131): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 1.6 MB/s | 845 kB 00:00 2023-03-03T06:15:10.543 INFO:teuthology.orchestra.run.smithi121.stdout:(2/131): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 966 kB/s | 845 kB 00:00 2023-03-03T06:15:10.630 INFO:teuthology.orchestra.run.smithi090.stdout:(3/131): ceph-immutable-object-cache-18.0.0-273 1.7 MB/s | 166 kB 00:00 2023-03-03T06:15:10.693 INFO:teuthology.orchestra.run.smithi121.stdout:(3/131): ceph-immutable-object-cache-18.0.0-273 1.1 MB/s | 166 kB 00:00 2023-03-03T06:15:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:(4/131): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 5.3 MB/s | 2.3 MB 00:00 2023-03-03T06:15:11.389 INFO:teuthology.orchestra.run.smithi090.stdout:(5/131): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 4.6 MB/s | 1.5 MB 00:00 2023-03-03T06:15:12.028 INFO:teuthology.orchestra.run.smithi121.stdout:(4/131): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 1.7 MB/s | 2.3 MB 00:01 2023-03-03T06:15:12.265 INFO:teuthology.orchestra.run.smithi090.stdout:(6/131): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 5.3 MB/s | 4.6 MB 00:00 2023-03-03T06:15:12.927 INFO:teuthology.orchestra.run.smithi121.stdout:(5/131): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 1.7 MB/s | 1.5 MB 00:00 2023-03-03T06:15:14.507 INFO:teuthology.orchestra.run.smithi090.stdout:(7/131): ceph-common-18.0.0-2739.g472aaef4.el8. 4.5 MB/s | 22 MB 00:04 2023-03-03T06:15:15.010 INFO:teuthology.orchestra.run.smithi121.stdout:(6/131): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 2.2 MB/s | 4.6 MB 00:02 2023-03-03T06:15:15.518 INFO:teuthology.orchestra.run.smithi121.stdout:(7/131): ceph-base-18.0.0-2739.g472aaef4.el8.x8 893 kB/s | 5.4 MB 00:06 2023-03-03T06:15:15.757 INFO:teuthology.orchestra.run.smithi090.stdout:(8/131): ceph-osd-18.0.0-2739.g472aaef4.el8.x86 5.1 MB/s | 18 MB 00:03 2023-03-03T06:15:15.816 INFO:teuthology.orchestra.run.smithi090.stdout:(9/131): ceph-selinux-18.0.0-2739.g472aaef4.el8 418 kB/s | 24 kB 00:00 2023-03-03T06:15:16.717 INFO:teuthology.orchestra.run.smithi090.stdout:(10/131): ceph-radosgw-18.0.0-2739.g472aaef4.el 5.9 MB/s | 13 MB 00:02 2023-03-03T06:15:16.808 INFO:teuthology.orchestra.run.smithi090.stdout:(11/131): cephfs-mirror-18.0.0-2739.g472aaef4.e 2.7 MB/s | 257 kB 00:00 2023-03-03T06:15:16.892 INFO:teuthology.orchestra.run.smithi090.stdout:(12/131): libcephfs-devel-18.0.0-2739.g472aaef4 366 kB/s | 30 kB 00:00 2023-03-03T06:15:17.084 INFO:teuthology.orchestra.run.smithi090.stdout:(13/131): libcephfs2-18.0.0-2739.g472aaef4.el8. 3.9 MB/s | 767 kB 00:00 2023-03-03T06:15:17.176 INFO:teuthology.orchestra.run.smithi090.stdout:(14/131): libcephsqlite-18.0.0-2739.g472aaef4.e 1.5 MB/s | 137 kB 00:00 2023-03-03T06:15:17.276 INFO:teuthology.orchestra.run.smithi090.stdout:(15/131): librados-devel-18.0.0-2739.g472aaef4. 1.4 MB/s | 139 kB 00:00 2023-03-03T06:15:17.427 INFO:teuthology.orchestra.run.smithi090.stdout:(16/131): libradosstriper1-18.0.0-2739.g472aaef 3.1 MB/s | 474 kB 00:00 2023-03-03T06:15:17.719 INFO:teuthology.orchestra.run.smithi090.stdout:(17/131): ceph-base-18.0.0-2739.g472aaef4.el8.x 684 kB/s | 5.4 MB 00:08 2023-03-03T06:15:17.802 INFO:teuthology.orchestra.run.smithi090.stdout:(18/131): python3-ceph-argparse-18.0.0-2739.g47 536 kB/s | 45 kB 00:00 2023-03-03T06:15:17.937 INFO:teuthology.orchestra.run.smithi090.stdout:(19/131): python3-ceph-common-18.0.0-2739.g472a 879 kB/s | 117 kB 00:00 2023-03-03T06:15:18.111 INFO:teuthology.orchestra.run.smithi090.stdout:(20/131): python3-cephfs-18.0.0-2739.g472aaef4. 1.1 MB/s | 190 kB 00:00 2023-03-03T06:15:18.362 INFO:teuthology.orchestra.run.smithi090.stdout:(21/131): python3-rados-18.0.0-2739.g472aaef4.e 1.4 MB/s | 371 kB 00:00 2023-03-03T06:15:18.454 INFO:teuthology.orchestra.run.smithi090.stdout:(22/131): librgw2-18.0.0-2739.g472aaef4.el8.x86 6.3 MB/s | 6.5 MB 00:01 2023-03-03T06:15:18.529 INFO:teuthology.orchestra.run.smithi090.stdout:(23/131): python3-rgw-18.0.0-2739.g472aaef4.el8 1.4 MB/s | 107 kB 00:00 2023-03-03T06:15:18.604 INFO:teuthology.orchestra.run.smithi090.stdout:(24/131): python3-rbd-18.0.0-2739.g472aaef4.el8 1.5 MB/s | 364 kB 00:00 2023-03-03T06:15:18.631 INFO:teuthology.orchestra.run.smithi090.stdout:(25/131): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 930 kB/s | 94 kB 00:00 2023-03-03T06:15:18.698 INFO:teuthology.orchestra.run.smithi090.stdout:(26/131): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 2.6 MB/s | 180 kB 00:00 2023-03-03T06:15:18.773 INFO:teuthology.orchestra.run.smithi090.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2739.g 308 kB/s | 23 kB 00:00 2023-03-03T06:15:18.840 INFO:teuthology.orchestra.run.smithi090.stdout:(28/131): ceph-mgr-cephadm-18.0.0-2739.g472aaef 1.8 MB/s | 123 kB 00:00 2023-03-03T06:15:19.116 INFO:teuthology.orchestra.run.smithi090.stdout:(29/131): ceph-mgr-dashboard-18.0.0-2739.g472aa 6.3 MB/s | 1.7 MB 00:00 2023-03-03T06:15:19.958 INFO:teuthology.orchestra.run.smithi090.stdout:(30/131): rbd-mirror-18.0.0-2739.g472aaef4.el8. 2.5 MB/s | 3.4 MB 00:01 2023-03-03T06:15:20.125 INFO:teuthology.orchestra.run.smithi090.stdout:(31/131): ceph-mgr-diskprediction-local-18.0.0- 7.3 MB/s | 7.4 MB 00:01 2023-03-03T06:15:20.150 INFO:teuthology.orchestra.run.smithi090.stdout:(32/131): ceph-mgr-modules-core-18.0.0-2739.g47 1.2 MB/s | 236 kB 00:00 2023-03-03T06:15:20.176 INFO:teuthology.orchestra.run.smithi090.stdout:(33/131): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 945 kB/s | 47 kB 00:00 2023-03-03T06:15:20.201 INFO:teuthology.orchestra.run.smithi090.stdout:(34/131): ceph-prometheus-alerts-18.0.0-2739.g4 289 kB/s | 15 kB 00:00 2023-03-03T06:15:20.234 INFO:teuthology.orchestra.run.smithi090.stdout:(35/131): ceph-volume-18.0.0-2739.g472aaef4.el8 4.4 MB/s | 263 kB 00:00 2023-03-03T06:15:20.318 INFO:teuthology.orchestra.run.smithi121.stdout:(8/131): ceph-osd-18.0.0-2739.g472aaef4.el8.x86 3.3 MB/s | 18 MB 00:05 2023-03-03T06:15:20.326 INFO:teuthology.orchestra.run.smithi090.stdout:(36/131): cephadm-18.0.0-2739.g472aaef4.el8.noa 665 kB/s | 83 kB 00:00 2023-03-03T06:15:20.376 INFO:teuthology.orchestra.run.smithi121.stdout:(9/131): ceph-selinux-18.0.0-2739.g472aaef4.el8 420 kB/s | 24 kB 00:00 2023-03-03T06:15:20.519 INFO:teuthology.orchestra.run.smithi121.stdout:(10/131): ceph-radosgw-18.0.0-2739.g472aaef4.el 2.6 MB/s | 13 MB 00:04 2023-03-03T06:15:20.568 INFO:teuthology.orchestra.run.smithi090.stdout:(37/131): cephfs-top-18.0.0-2739.g472aaef4.el8. 54 kB/s | 18 kB 00:00 2023-03-03T06:15:20.647 INFO:teuthology.orchestra.run.smithi121.stdout:(11/131): cephfs-mirror-18.0.0-2739.g472aaef4.e 2.0 MB/s | 257 kB 00:00 2023-03-03T06:15:20.702 INFO:teuthology.orchestra.run.smithi090.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 441 kB/s | 59 kB 00:00 2023-03-03T06:15:20.727 INFO:teuthology.orchestra.run.smithi121.stdout:(12/131): libcephfs-devel-18.0.0-2739.g472aaef4 378 kB/s | 30 kB 00:00 2023-03-03T06:15:20.761 INFO:teuthology.orchestra.run.smithi090.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 705 kB/s | 306 kB 00:00 2023-03-03T06:15:20.786 INFO:teuthology.orchestra.run.smithi090.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 897 kB/s | 75 kB 00:00 2023-03-03T06:15:20.837 INFO:teuthology.orchestra.run.smithi090.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 871 kB/s | 44 kB 00:00 2023-03-03T06:15:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.3 MB/s | 476 kB 00:00 2023-03-03T06:15:20.903 INFO:teuthology.orchestra.run.smithi121.stdout:(13/131): libcephfs2-18.0.0-2739.g472aaef4.el8. 4.3 MB/s | 767 kB 00:00 2023-03-03T06:15:20.913 INFO:teuthology.orchestra.run.smithi090.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.4 MB/s | 185 kB 00:00 2023-03-03T06:15:20.938 INFO:teuthology.orchestra.run.smithi090.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 492 kB/s | 33 kB 00:00 2023-03-03T06:15:20.963 INFO:teuthology.orchestra.run.smithi090.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2023-03-03T06:15:20.992 INFO:teuthology.orchestra.run.smithi121.stdout:(14/131): libcephsqlite-18.0.0-2739.g472aaef4.e 1.5 MB/s | 137 kB 00:00 2023-03-03T06:15:20.997 INFO:teuthology.orchestra.run.smithi090.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2023-03-03T06:15:21.045 INFO:teuthology.orchestra.run.smithi121.stdout:(15/131): librados-devel-18.0.0-2739.g472aaef4. 2.6 MB/s | 139 kB 00:00 2023-03-03T06:15:21.047 INFO:teuthology.orchestra.run.smithi090.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 4.5 MB/s | 384 kB 00:00 2023-03-03T06:15:21.072 INFO:teuthology.orchestra.run.smithi090.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2023-03-03T06:15:21.099 INFO:teuthology.orchestra.run.smithi090.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 215 kB/s | 11 kB 00:00 2023-03-03T06:15:21.124 INFO:teuthology.orchestra.run.smithi090.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 342 kB/s | 18 kB 00:00 2023-03-03T06:15:21.170 INFO:teuthology.orchestra.run.smithi121.stdout:(16/131): libradosstriper1-18.0.0-2739.g472aaef 3.7 MB/s | 474 kB 00:00 2023-03-03T06:15:21.174 INFO:teuthology.orchestra.run.smithi090.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 973 kB/s | 49 kB 00:00 2023-03-03T06:15:21.233 INFO:teuthology.orchestra.run.smithi090.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-03-03T06:15:21.275 INFO:teuthology.orchestra.run.smithi090.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 6.6 MB/s | 1.2 MB 00:00 2023-03-03T06:15:21.301 INFO:teuthology.orchestra.run.smithi090.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 902 kB/s | 60 kB 00:00 2023-03-03T06:15:21.326 INFO:teuthology.orchestra.run.smithi090.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-03-03T06:15:21.351 INFO:teuthology.orchestra.run.smithi090.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-03-03T06:15:21.402 INFO:teuthology.orchestra.run.smithi090.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 447 kB/s | 34 kB 00:00 2023-03-03T06:15:21.427 INFO:teuthology.orchestra.run.smithi090.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 2.6 MB/s | 196 kB 00:00 2023-03-03T06:15:21.461 INFO:teuthology.orchestra.run.smithi090.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2023-03-03T06:15:21.486 INFO:teuthology.orchestra.run.smithi090.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 319 kB/s | 19 kB 00:00 2023-03-03T06:15:21.511 INFO:teuthology.orchestra.run.smithi090.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2023-03-03T06:15:21.546 INFO:teuthology.orchestra.run.smithi090.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 498 kB/s | 29 kB 00:00 2023-03-03T06:15:21.570 INFO:teuthology.orchestra.run.smithi090.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 469 kB/s | 27 kB 00:00 2023-03-03T06:15:21.604 INFO:teuthology.orchestra.run.smithi090.stdout:(64/131): python3-waitress-1.2.1-2.el8.1.noarch 4.0 MB/s | 241 kB 00:00 2023-03-03T06:15:21.632 INFO:teuthology.orchestra.run.smithi090.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2023-03-03T06:15:21.655 INFO:teuthology.orchestra.run.smithi090.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-03-03T06:15:21.680 INFO:teuthology.orchestra.run.smithi090.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-03-03T06:15:21.687 INFO:teuthology.orchestra.run.smithi121.stdout:(17/131): ceph-common-18.0.0-2739.g472aaef4.el8 1.7 MB/s | 22 MB 00:12 2023-03-03T06:15:21.705 INFO:teuthology.orchestra.run.smithi090.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 457 kB/s | 23 kB 00:00 2023-03-03T06:15:21.730 INFO:teuthology.orchestra.run.smithi090.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-03-03T06:15:21.746 INFO:teuthology.orchestra.run.smithi121.stdout:(18/131): python3-ceph-argparse-18.0.0-2739.g47 770 kB/s | 45 kB 00:00 2023-03-03T06:15:21.781 INFO:teuthology.orchestra.run.smithi090.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 937 kB/s | 47 kB 00:00 2023-03-03T06:15:21.804 INFO:teuthology.orchestra.run.smithi121.stdout:(19/131): python3-ceph-common-18.0.0-2739.g472a 2.0 MB/s | 117 kB 00:00 2023-03-03T06:15:21.880 INFO:teuthology.orchestra.run.smithi121.stdout:(20/131): python3-cephfs-18.0.0-2739.g472aaef4. 2.5 MB/s | 190 kB 00:00 2023-03-03T06:15:21.971 INFO:teuthology.orchestra.run.smithi121.stdout:(21/131): python3-rados-18.0.0-2739.g472aaef4.e 4.0 MB/s | 371 kB 00:00 2023-03-03T06:15:22.055 INFO:teuthology.orchestra.run.smithi121.stdout:(22/131): python3-rbd-18.0.0-2739.g472aaef4.el8 4.3 MB/s | 364 kB 00:00 2023-03-03T06:15:22.164 INFO:teuthology.orchestra.run.smithi121.stdout:(23/131): librgw2-18.0.0-2739.g472aaef4.el8.x86 6.5 MB/s | 6.5 MB 00:00 2023-03-03T06:15:22.189 INFO:teuthology.orchestra.run.smithi121.stdout:(24/131): python3-rgw-18.0.0-2739.g472aaef4.el8 800 kB/s | 107 kB 00:00 2023-03-03T06:15:22.214 INFO:teuthology.orchestra.run.smithi121.stdout:(25/131): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 1.8 MB/s | 94 kB 00:00 2023-03-03T06:15:22.281 INFO:teuthology.orchestra.run.smithi121.stdout:(26/131): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 2.6 MB/s | 180 kB 00:00 2023-03-03T06:15:22.298 INFO:teuthology.orchestra.run.smithi090.stdout:(71/131): ceph-test-18.0.0-2739.g472aaef4.el8.x 7.2 MB/s | 47 MB 00:06 2023-03-03T06:15:22.323 INFO:teuthology.orchestra.run.smithi121.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2739.g 552 kB/s | 23 kB 00:00 2023-03-03T06:15:22.332 INFO:teuthology.orchestra.run.smithi090.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 363 kB/s | 200 kB 00:00 2023-03-03T06:15:22.366 INFO:teuthology.orchestra.run.smithi121.stdout:(28/131): ceph-mgr-cephadm-18.0.0-2739.g472aaef 2.9 MB/s | 123 kB 00:00 2023-03-03T06:15:22.382 INFO:teuthology.orchestra.run.smithi090.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2023-03-03T06:15:22.409 INFO:teuthology.orchestra.run.smithi090.stdout:(74/131): python3-jwt-1.6.1-2.el8.noarch.rpm 388 kB/s | 43 kB 00:00 2023-03-03T06:15:22.442 INFO:teuthology.orchestra.run.smithi090.stdout:(75/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2023-03-03T06:15:22.467 INFO:teuthology.orchestra.run.smithi090.stdout:(76/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 565 kB/s | 48 kB 00:00 2023-03-03T06:15:22.493 INFO:teuthology.orchestra.run.smithi090.stdout:(77/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2023-03-03T06:15:22.518 INFO:teuthology.orchestra.run.smithi090.stdout:(78/131): python3-requests-oauthlib-1.0.0-1.el8 564 kB/s | 43 kB 00:00 2023-03-03T06:15:22.552 INFO:teuthology.orchestra.run.smithi090.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 1.3 MB/s | 109 kB 00:00 2023-03-03T06:15:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.7 MB/s | 229 kB 00:00 2023-03-03T06:15:22.597 INFO:teuthology.orchestra.run.smithi121.stdout:(29/131): rbd-mirror-18.0.0-2739.g472aaef4.el8. 8.5 MB/s | 3.4 MB 00:00 2023-03-03T06:15:22.602 INFO:teuthology.orchestra.run.smithi090.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 372 kB/s | 19 kB 00:00 2023-03-03T06:15:22.636 INFO:teuthology.orchestra.run.smithi090.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.6 MB/s | 155 kB 00:00 2023-03-03T06:15:22.641 INFO:teuthology.orchestra.run.smithi121.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2739.g472aa 6.3 MB/s | 1.7 MB 00:00 2023-03-03T06:15:22.678 INFO:teuthology.orchestra.run.smithi090.stdout:(83/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.4 MB/s | 560 kB 00:00 2023-03-03T06:15:22.700 INFO:teuthology.orchestra.run.smithi121.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2739.g47 4.0 MB/s | 236 kB 00:00 2023-03-03T06:15:22.703 INFO:teuthology.orchestra.run.smithi090.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 267 kB/s | 18 kB 00:00 2023-03-03T06:15:22.737 INFO:teuthology.orchestra.run.smithi090.stdout:(85/131): python3-cryptography-3.2.1-5.el8.x86_ 4.1 MB/s | 559 kB 00:00 2023-03-03T06:15:22.758 INFO:teuthology.orchestra.run.smithi121.stdout:(32/131): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 817 kB/s | 47 kB 00:00 2023-03-03T06:15:22.771 INFO:teuthology.orchestra.run.smithi090.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-03-03T06:15:22.800 INFO:teuthology.orchestra.run.smithi121.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2739.g4 349 kB/s | 15 kB 00:00 2023-03-03T06:15:22.804 INFO:teuthology.orchestra.run.smithi090.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2023-03-03T06:15:22.830 INFO:teuthology.orchestra.run.smithi090.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2023-03-03T06:15:22.855 INFO:teuthology.orchestra.run.smithi090.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-03-03T06:15:22.867 INFO:teuthology.orchestra.run.smithi121.stdout:(34/131): ceph-volume-18.0.0-2739.g472aaef4.el8 3.9 MB/s | 263 kB 00:00 2023-03-03T06:15:22.880 INFO:teuthology.orchestra.run.smithi090.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-03-03T06:15:22.906 INFO:teuthology.orchestra.run.smithi090.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-03-03T06:15:22.931 INFO:teuthology.orchestra.run.smithi090.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-03-03T06:15:22.934 INFO:teuthology.orchestra.run.smithi121.stdout:(35/131): cephadm-18.0.0-2739.g472aaef4.el8.noa 1.2 MB/s | 83 kB 00:00 2023-03-03T06:15:22.979 INFO:teuthology.orchestra.run.smithi090.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 531 kB/s | 39 kB 00:00 2023-03-03T06:15:22.992 INFO:teuthology.orchestra.run.smithi121.stdout:(36/131): cephfs-top-18.0.0-2739.g472aaef4.el8. 310 kB/s | 18 kB 00:00 2023-03-03T06:15:23.024 INFO:teuthology.orchestra.run.smithi090.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 6.8 MB/s | 644 kB 00:00 2023-03-03T06:15:23.051 INFO:teuthology.orchestra.run.smithi090.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.4 MB/s | 171 kB 00:00 2023-03-03T06:15:23.074 INFO:teuthology.orchestra.run.smithi090.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-03-03T06:15:23.141 INFO:teuthology.orchestra.run.smithi090.stdout:(97/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 5.8 MB/s | 2.3 MB 00:00 2023-03-03T06:15:23.167 INFO:teuthology.orchestra.run.smithi090.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.2 MB/s | 110 kB 00:00 2023-03-03T06:15:23.176 INFO:teuthology.orchestra.run.smithi121.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-03-03T06:15:23.193 INFO:teuthology.orchestra.run.smithi090.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.0 MB/s | 54 kB 00:00 2023-03-03T06:15:23.218 INFO:teuthology.orchestra.run.smithi090.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.9 MB/s | 259 kB 00:00 2023-03-03T06:15:23.251 INFO:teuthology.orchestra.run.smithi121.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 785 kB/s | 59 kB 00:00 2023-03-03T06:15:23.252 INFO:teuthology.orchestra.run.smithi090.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-03-03T06:15:23.277 INFO:teuthology.orchestra.run.smithi090.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 663 kB/s | 39 kB 00:00 2023-03-03T06:15:23.319 INFO:teuthology.orchestra.run.smithi090.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-03-03T06:15:23.360 INFO:teuthology.orchestra.run.smithi121.stdout:(39/131): ceph-mgr-diskprediction-local-18.0.0- 9.7 MB/s | 7.4 MB 00:00 2023-03-03T06:15:23.361 INFO:teuthology.orchestra.run.smithi090.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 6.3 MB/s | 538 kB 00:00 2023-03-03T06:15:23.386 INFO:teuthology.orchestra.run.smithi121.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 562 kB/s | 75 kB 00:00 2023-03-03T06:15:23.395 INFO:teuthology.orchestra.run.smithi090.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2023-03-03T06:15:23.420 INFO:teuthology.orchestra.run.smithi090.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2023-03-03T06:15:23.437 INFO:teuthology.orchestra.run.smithi121.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 874 kB/s | 44 kB 00:00 2023-03-03T06:15:23.471 INFO:teuthology.orchestra.run.smithi090.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-03-03T06:15:23.486 INFO:teuthology.orchestra.run.smithi121.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2023-03-03T06:15:23.504 INFO:teuthology.orchestra.run.smithi090.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 794 kB/s | 27 kB 00:00 2023-03-03T06:15:23.555 INFO:teuthology.orchestra.run.smithi090.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-03-03T06:15:23.557 INFO:teuthology.orchestra.run.smithi121.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 465 kB/s | 33 kB 00:00 2023-03-03T06:15:23.589 INFO:teuthology.orchestra.run.smithi090.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-03-03T06:15:23.595 INFO:teuthology.orchestra.run.smithi121.stdout:(44/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.0 MB/s | 476 kB 00:00 2023-03-03T06:15:23.620 INFO:teuthology.orchestra.run.smithi121.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 261 kB/s | 16 kB 00:00 2023-03-03T06:15:23.646 INFO:teuthology.orchestra.run.smithi121.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2023-03-03T06:15:23.696 INFO:teuthology.orchestra.run.smithi121.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-03-03T06:15:23.697 INFO:teuthology.orchestra.run.smithi090.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-03-03T06:15:23.729 INFO:teuthology.orchestra.run.smithi121.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2023-03-03T06:15:23.748 INFO:teuthology.orchestra.run.smithi090.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-03-03T06:15:23.755 INFO:teuthology.orchestra.run.smithi121.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2023-03-03T06:15:23.780 INFO:teuthology.orchestra.run.smithi121.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2023-03-03T06:15:23.797 INFO:teuthology.orchestra.run.smithi090.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 4.4 MB/s | 225 kB 00:00 2023-03-03T06:15:23.830 INFO:teuthology.orchestra.run.smithi121.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 974 kB/s | 49 kB 00:00 2023-03-03T06:15:23.873 INFO:teuthology.orchestra.run.smithi090.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 13 MB/s | 3.7 MB 00:00 2023-03-03T06:15:23.898 INFO:teuthology.orchestra.run.smithi090.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 638 kB/s | 64 kB 00:00 2023-03-03T06:15:23.906 INFO:teuthology.orchestra.run.smithi121.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 785 kB/s | 59 kB 00:00 2023-03-03T06:15:23.924 INFO:teuthology.orchestra.run.smithi090.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-03-03T06:15:23.949 INFO:teuthology.orchestra.run.smithi090.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 396 kB/s | 20 kB 00:00 2023-03-03T06:15:23.956 INFO:teuthology.orchestra.run.smithi121.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 5.8 MB/s | 1.2 MB 00:00 2023-03-03T06:15:23.974 INFO:teuthology.orchestra.run.smithi090.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-03-03T06:15:23.981 INFO:teuthology.orchestra.run.smithi121.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 802 kB/s | 60 kB 00:00 2023-03-03T06:15:24.015 INFO:teuthology.orchestra.run.smithi121.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2023-03-03T06:15:24.016 INFO:teuthology.orchestra.run.smithi090.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-03-03T06:15:24.032 INFO:teuthology.orchestra.run.smithi121.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-03-03T06:15:24.065 INFO:teuthology.orchestra.run.smithi121.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 670 kB/s | 34 kB 00:00 2023-03-03T06:15:24.091 INFO:teuthology.orchestra.run.smithi121.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2023-03-03T06:15:24.108 INFO:teuthology.orchestra.run.smithi121.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.4 MB/s | 61 kB 00:00 2023-03-03T06:15:24.133 INFO:teuthology.orchestra.run.smithi121.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 447 kB/s | 19 kB 00:00 2023-03-03T06:15:24.150 INFO:teuthology.orchestra.run.smithi121.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 573 kB/s | 24 kB 00:00 2023-03-03T06:15:24.183 INFO:teuthology.orchestra.run.smithi090.stdout:(120/131): python3-scipy-1.0.0-21.module+el8.5. 12 MB/s | 14 MB 00:01 2023-03-03T06:15:24.206 INFO:teuthology.orchestra.run.smithi121.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 409 kB/s | 29 kB 00:00 2023-03-03T06:15:24.208 INFO:teuthology.orchestra.run.smithi090.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.0 MB/s | 198 kB 00:00 2023-03-03T06:15:24.226 INFO:teuthology.orchestra.run.smithi121.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 366 kB/s | 27 kB 00:00 2023-03-03T06:15:24.251 INFO:teuthology.orchestra.run.smithi090.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-03-03T06:15:24.259 INFO:teuthology.orchestra.run.smithi121.stdout:(64/131): python3-waitress-1.2.1-2.el8.1.noarch 4.4 MB/s | 241 kB 00:00 2023-03-03T06:15:24.284 INFO:teuthology.orchestra.run.smithi121.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2023-03-03T06:15:24.301 INFO:teuthology.orchestra.run.smithi090.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-03-03T06:15:24.301 INFO:teuthology.orchestra.run.smithi121.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2023-03-03T06:15:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-03-03T06:15:24.352 INFO:teuthology.orchestra.run.smithi121.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 456 kB/s | 23 kB 00:00 2023-03-03T06:15:24.369 INFO:teuthology.orchestra.run.smithi121.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-03-03T06:15:24.402 INFO:teuthology.orchestra.run.smithi090.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 11 MB/s | 4.9 MB 00:00 2023-03-03T06:15:24.411 INFO:teuthology.orchestra.run.smithi121.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-03-03T06:15:24.453 INFO:teuthology.orchestra.run.smithi121.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-03-03T06:15:24.486 INFO:teuthology.orchestra.run.smithi121.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-03-03T06:15:24.503 INFO:teuthology.orchestra.run.smithi090.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-03-03T06:15:24.545 INFO:teuthology.orchestra.run.smithi090.stdout:(126/131): postgresql-contrib-10.21-2.module+el 5.6 MB/s | 808 kB 00:00 2023-03-03T06:15:24.553 INFO:teuthology.orchestra.run.smithi121.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2023-03-03T06:15:24.587 INFO:teuthology.orchestra.run.smithi121.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 6.9 MB/s | 1.7 MB 00:00 2023-03-03T06:15:24.604 INFO:teuthology.orchestra.run.smithi090.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 15 MB/s | 1.5 MB 00:00 2023-03-03T06:15:24.612 INFO:teuthology.orchestra.run.smithi121.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 822 kB/s | 48 kB 00:00 2023-03-03T06:15:24.654 INFO:teuthology.orchestra.run.smithi121.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.2 MB/s | 43 kB 00:00 2023-03-03T06:15:24.679 INFO:teuthology.orchestra.run.smithi121.stdout:(77/131): libsepol-devel-2.9-3.el8.x86_64.rpm 945 kB/s | 87 kB 00:00 2023-03-03T06:15:24.704 INFO:teuthology.orchestra.run.smithi090.stdout:(128/131): postgresql-server-10.21-2.module+el8 13 MB/s | 5.1 MB 00:00 2023-03-03T06:15:24.705 INFO:teuthology.orchestra.run.smithi121.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-03-03T06:15:24.730 INFO:teuthology.orchestra.run.smithi121.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-03-03T06:15:24.754 INFO:teuthology.orchestra.run.smithi090.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-03T06:15:24.763 INFO:teuthology.orchestra.run.smithi121.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-03-03T06:15:24.789 INFO:teuthology.orchestra.run.smithi121.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 319 kB/s | 19 kB 00:00 2023-03-03T06:15:24.814 INFO:teuthology.orchestra.run.smithi121.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-03-03T06:15:24.864 INFO:teuthology.orchestra.run.smithi121.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 7.3 MB/s | 559 kB 00:00 2023-03-03T06:15:24.889 INFO:teuthology.orchestra.run.smithi121.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 238 kB/s | 18 kB 00:00 2023-03-03T06:15:24.925 INFO:teuthology.orchestra.run.smithi121.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 9.8 MB/s | 605 kB 00:00 2023-03-03T06:15:24.950 INFO:teuthology.orchestra.run.smithi121.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.8 MB/s | 238 kB 00:00 2023-03-03T06:15:24.992 INFO:teuthology.orchestra.run.smithi121.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-03-03T06:15:25.043 INFO:teuthology.orchestra.run.smithi121.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 20 MB/s | 2.3 MB 00:00 2023-03-03T06:15:25.068 INFO:teuthology.orchestra.run.smithi121.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-03-03T06:15:25.093 INFO:teuthology.orchestra.run.smithi121.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-03-03T06:15:25.118 INFO:teuthology.orchestra.run.smithi121.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-03-03T06:15:25.168 INFO:teuthology.orchestra.run.smithi121.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 781 kB/s | 58 kB 00:00 2023-03-03T06:15:25.194 INFO:teuthology.orchestra.run.smithi121.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 512 kB/s | 39 kB 00:00 2023-03-03T06:15:25.227 INFO:teuthology.orchestra.run.smithi121.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 11 MB/s | 644 kB 00:00 2023-03-03T06:15:25.253 INFO:teuthology.orchestra.run.smithi121.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.9 MB/s | 171 kB 00:00 2023-03-03T06:15:25.278 INFO:teuthology.orchestra.run.smithi121.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-03-03T06:15:25.312 INFO:teuthology.orchestra.run.smithi121.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 3.2 MB/s | 110 kB 00:00 2023-03-03T06:15:25.337 INFO:teuthology.orchestra.run.smithi121.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-03-03T06:15:25.379 INFO:teuthology.orchestra.run.smithi121.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-03-03T06:15:25.429 INFO:teuthology.orchestra.run.smithi121.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-03-03T06:15:25.546 INFO:teuthology.orchestra.run.smithi121.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 333 kB/s | 39 kB 00:00 2023-03-03T06:15:25.597 INFO:teuthology.orchestra.run.smithi121.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 14 MB/s | 688 kB 00:00 2023-03-03T06:15:25.638 INFO:teuthology.orchestra.run.smithi121.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 13 MB/s | 538 kB 00:00 2023-03-03T06:15:25.672 INFO:teuthology.orchestra.run.smithi121.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 4.6 MB/s | 157 kB 00:00 2023-03-03T06:15:25.715 INFO:teuthology.orchestra.run.smithi121.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-03-03T06:15:25.881 INFO:teuthology.orchestra.run.smithi121.stdout:(106/131): python3-scipy-1.0.0-21.module+el8.5. 22 MB/s | 14 MB 00:00 2023-03-03T06:15:25.923 INFO:teuthology.orchestra.run.smithi121.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 3.3 MB/s | 713 kB 00:00 2023-03-03T06:15:25.956 INFO:teuthology.orchestra.run.smithi121.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 797 kB/s | 27 kB 00:00 2023-03-03T06:15:26.007 INFO:teuthology.orchestra.run.smithi121.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-03-03T06:15:26.041 INFO:teuthology.orchestra.run.smithi121.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-03-03T06:15:26.115 INFO:teuthology.orchestra.run.smithi121.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-03-03T06:15:26.157 INFO:teuthology.orchestra.run.smithi121.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2023-03-03T06:15:26.208 INFO:teuthology.orchestra.run.smithi121.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 4.4 MB/s | 225 kB 00:00 2023-03-03T06:15:26.274 INFO:teuthology.orchestra.run.smithi121.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 16 MB/s | 3.7 MB 00:00 2023-03-03T06:15:26.308 INFO:teuthology.orchestra.run.smithi121.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 637 kB/s | 64 kB 00:00 2023-03-03T06:15:26.334 INFO:teuthology.orchestra.run.smithi121.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.1 MB/s | 126 kB 00:00 2023-03-03T06:15:26.359 INFO:teuthology.orchestra.run.smithi121.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 397 kB/s | 20 kB 00:00 2023-03-03T06:15:26.384 INFO:teuthology.orchestra.run.smithi121.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-03-03T06:15:26.418 INFO:teuthology.orchestra.run.smithi121.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-03-03T06:15:26.460 INFO:teuthology.orchestra.run.smithi121.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-03-03T06:15:26.560 INFO:teuthology.orchestra.run.smithi121.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 28 MB/s | 4.9 MB 00:00 2023-03-03T06:15:26.602 INFO:teuthology.orchestra.run.smithi121.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-03T06:15:26.669 INFO:teuthology.orchestra.run.smithi121.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-03-03T06:15:26.694 INFO:teuthology.orchestra.run.smithi121.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.8 MB/s | 354 kB 00:00 2023-03-03T06:15:26.753 INFO:teuthology.orchestra.run.smithi121.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-03-03T06:15:26.843 INFO:teuthology.orchestra.run.smithi121.stdout:(126/131): postgresql-10.21-2.module+el8.6.0+15 17 MB/s | 1.5 MB 00:00 2023-03-03T06:15:26.920 INFO:teuthology.orchestra.run.smithi121.stdout:(127/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-03-03T06:15:31.396 INFO:teuthology.orchestra.run.smithi090.stdout:(130/131): librados2-18.0.0-2739.g472aaef4.el8. 546 kB/s | 3.7 MB 00:06 2023-03-03T06:15:31.453 INFO:teuthology.orchestra.run.smithi121.stdout:(128/131): librbd1-18.0.0-2739.g472aaef4.el8.x8 804 kB/s | 3.6 MB 00:04 2023-03-03T06:15:31.520 INFO:teuthology.orchestra.run.smithi121.stdout:(129/131): librados2-18.0.0-2739.g472aaef4.el8. 801 kB/s | 3.7 MB 00:04 2023-03-03T06:15:31.553 INFO:teuthology.orchestra.run.smithi121.stdout:(130/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 7.8 MB/s | 798 kB 00:00 2023-03-03T06:15:31.613 INFO:teuthology.orchestra.run.smithi090.stdout:(131/131): librbd1-18.0.0-2739.g472aaef4.el8.x8 520 kB/s | 3.6 MB 00:07 2023-03-03T06:15:31.616 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2023-03-03T06:15:31.616 INFO:teuthology.orchestra.run.smithi090.stdout:Total 9.2 MB/s | 203 MB 00:21 2023-03-03T06:15:32.472 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2023-03-03T06:15:32.592 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2023-03-03T06:15:32.592 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2023-03-03T06:15:32.762 INFO:teuthology.orchestra.run.smithi121.stdout:(131/131): ceph-test-18.0.0-2739.g472aaef4.el8. 3.8 MB/s | 47 MB 00:12 2023-03-03T06:15:32.765 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-03-03T06:15:32.765 INFO:teuthology.orchestra.run.smithi121.stdout:Total 8.7 MB/s | 203 MB 00:23 2023-03-03T06:15:33.598 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-03-03T06:15:33.680 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2023-03-03T06:15:33.685 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2023-03-03T06:15:33.718 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-03-03T06:15:33.719 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-03-03T06:15:34.841 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-03-03T06:15:34.845 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-03-03T06:15:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2023-03-03T06:15:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-03T06:15:39.300 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-03T06:15:39.461 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-03T06:15:39.582 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-03T06:15:39.778 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-03T06:15:40.177 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-03T06:15:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-03-03T06:15:40.419 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 5/134 2023-03-03T06:15:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-03T06:15:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-03T06:15:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 5/134 2023-03-03T06:15:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-03T06:15:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-03T06:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T06:15:41.165 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T06:15:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-03T06:15:41.361 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T06:15:41.458 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T06:15:41.606 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-03T06:15:41.633 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T06:15:41.851 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-03-03T06:15:41.893 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 5/134 2023-03-03T06:15:42.093 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-03T06:15:42.285 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 5/134 2023-03-03T06:15:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-03T06:15:42.526 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T06:15:42.652 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T06:15:42.760 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-03T06:15:42.876 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T06:15:42.974 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T06:15:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-03T06:15:43.153 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T06:15:43.164 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T06:15:43.264 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T06:15:43.343 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-03-03T06:15:43.444 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T06:15:43.595 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T06:15:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-03T06:15:43.798 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-03-03T06:15:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-03T06:15:43.975 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-03-03T06:15:44.132 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-03-03T06:15:44.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-03T06:15:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-03T06:15:44.405 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-03T06:15:44.552 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-03T06:15:44.596 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-03T06:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T06:15:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T06:15:45.109 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T06:15:45.165 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-03T06:15:45.251 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T06:15:45.419 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-03T06:15:45.479 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-03-03T06:15:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-03-03T06:15:45.886 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-03-03T06:15:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-03T06:15:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-03T06:15:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-03T06:15:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-03-03T06:15:46.359 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-03T06:15:46.955 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-03-03T06:15:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-03T06:15:47.215 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-03-03T06:15:47.217 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-03T06:15:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-03-03T06:15:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-03T06:15:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-03T06:15:48.041 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-03-03T06:15:48.084 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 26/134 2023-03-03T06:15:48.213 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 27/134 2023-03-03T06:15:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 28/134 2023-03-03T06:15:48.677 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-03T06:15:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-03-03T06:15:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-03-03T06:15:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-03-03T06:15:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-03T06:15:49.706 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 26/134 2023-03-03T06:15:49.843 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 27/134 2023-03-03T06:15:50.098 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 28/134 2023-03-03T06:15:50.308 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-03T06:15:50.457 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-03T06:15:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-03-03T06:15:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-03-03T06:15:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-03-03T06:15:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-03T06:15:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-03-03T06:15:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-03T06:15:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-03T06:15:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 36/134 2023-03-03T06:15:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 37/134 2023-03-03T06:15:53.159 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-03-03T06:15:53.321 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-03-03T06:15:53.334 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-03-03T06:15:53.502 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-03-03T06:15:53.572 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-03-03T06:15:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-03-03T06:15:53.789 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-03-03T06:15:53.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-03-03T06:15:53.991 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-03-03T06:15:54.136 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-03T06:15:54.156 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-03-03T06:15:54.218 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-03T06:15:54.409 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-03T06:15:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-03-03T06:15:54.608 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-03T06:15:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-03-03T06:15:54.780 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-03-03T06:15:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 36/134 2023-03-03T06:15:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-03-03T06:15:54.988 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 37/134 2023-03-03T06:15:55.138 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-03-03T06:15:55.147 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-03-03T06:15:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-03-03T06:15:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-03-03T06:15:55.468 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-03-03T06:15:55.482 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-03-03T06:15:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-03-03T06:15:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-03-03T06:15:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-03-03T06:15:56.017 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-03-03T06:15:56.043 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-03-03T06:15:56.248 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-03T06:15:56.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-03-03T06:15:56.325 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-03T06:15:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-03-03T06:15:56.553 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-03-03T06:15:56.718 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-03-03T06:15:56.749 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-03-03T06:15:56.907 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-03-03T06:15:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-03-03T06:15:57.095 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-03-03T06:15:57.270 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-03-03T06:15:57.317 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-03-03T06:15:57.467 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-03-03T06:15:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-03-03T06:15:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-03-03T06:15:57.703 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-03-03T06:15:57.940 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-03-03T06:15:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-03-03T06:15:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-03-03T06:15:58.299 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-03-03T06:15:58.299 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-03-03T06:15:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-03-03T06:15:58.624 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-03-03T06:15:58.841 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-03-03T06:15:58.864 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-03-03T06:15:59.080 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-03-03T06:15:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-03-03T06:15:59.312 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-03-03T06:15:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-03-03T06:15:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-03T06:15:59.663 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-03T06:15:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-03-03T06:15:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-03T06:15:59.988 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-03-03T06:16:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-03T06:16:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-03-03T06:16:00.300 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-03T06:16:00.463 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-03T06:16:00.477 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-03-03T06:16:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-03T06:16:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-03-03T06:16:00.832 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-03T06:16:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-03-03T06:16:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-03T06:16:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-03T06:16:01.237 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-03-03T06:16:01.316 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-03T06:16:01.538 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-03-03T06:16:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-03T06:16:01.773 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-03T06:16:01.801 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-03-03T06:16:01.950 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-03T06:16:02.107 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-03-03T06:16:02.208 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-03T06:16:02.279 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-03T06:16:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-03T06:16:02.483 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-03T06:16:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-03T06:16:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-03T06:16:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-03T06:16:02.978 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-03T06:16:03.151 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-03T06:16:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-03T06:16:03.523 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-03T06:16:03.718 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-03T06:16:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-03T06:16:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-03T06:16:04.147 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-03T06:16:04.150 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-03T06:16:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-03T06:16:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-03T06:16:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-03T06:16:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-03T06:16:04.817 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-03T06:16:04.898 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-03T06:16:05.002 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-03T06:16:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-03T06:16:05.285 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-03T06:16:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-03T06:16:05.728 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-03T06:16:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-03T06:16:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-03T06:16:06.077 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T06:16:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T06:16:06.361 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 93/134 2023-03-03T06:16:07.066 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-03T06:16:07.308 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-03T06:16:07.587 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-03T06:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-03T06:16:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-03T06:16:08.351 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-03T06:16:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-03T06:16:09.345 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T06:16:09.422 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T06:16:09.637 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 93/134 2023-03-03T06:16:09.839 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T06:16:10.069 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T06:16:10.655 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T06:16:10.926 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T06:16:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T06:16:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T06:16:13.210 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T06:16:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T06:16:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T06:16:14.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T06:16:14.443 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T06:16:14.619 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T06:16:30.514 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T06:16:30.514 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2023-03-03T06:16:30.514 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2023-03-03T06:16:30.515 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2023-03-03T06:16:30.515 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /run 2023-03-03T06:16:30.515 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2023-03-03T06:16:30.515 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2023-03-03T06:16:30.515 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2023-03-03T06:16:30.515 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2023-03-03T06:16:30.515 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2023-03-03T06:16:30.516 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:16:30.826 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T06:16:32.429 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T06:16:32.608 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T06:16:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T06:16:33.009 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T06:16:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T06:16:33.321 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2023-03-03T06:16:33.321 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2023-03-03T06:16:33.321 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-03-03T06:16:33.321 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2023-03-03T06:16:33.321 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2023-03-03T06:16:33.321 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2023-03-03T06:16:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2023-03-03T06:16:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2023-03-03T06:16:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-03-03T06:16:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:16:33.456 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T06:16:33.667 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T06:16:33.684 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T06:16:33.690 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T06:16:35.249 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T06:16:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T06:16:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T06:16:35.452 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T06:16:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 102/134 2023-03-03T06:16:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T06:16:35.748 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 103/134 2023-03-03T06:16:35.855 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T06:16:35.925 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T06:16:36.158 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T06:16:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T06:16:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T06:16:36.539 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T06:16:36.556 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T06:16:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T06:16:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T06:16:37.723 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T06:16:38.034 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 107/134 2023-03-03T06:16:38.116 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T06:16:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T06:16:38.301 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T06:16:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T06:16:38.461 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 102/134 2023-03-03T06:16:38.475 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T06:16:38.488 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T06:16:38.644 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 103/134 2023-03-03T06:16:38.855 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T06:16:39.071 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T06:16:39.378 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T06:16:39.736 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 110/134 2023-03-03T06:16:39.980 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T06:16:40.013 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T06:16:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T06:16:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T06:16:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T06:16:40.376 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T06:16:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T06:16:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T06:16:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T06:16:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T06:16:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 107/134 2023-03-03T06:16:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T06:16:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T06:16:41.480 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T06:16:41.491 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T06:16:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 110/134 2023-03-03T06:16:43.058 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T06:16:43.091 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T06:16:43.263 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T06:16:43.304 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T06:16:43.472 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T06:16:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T06:16:44.030 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T06:16:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T06:16:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 115/134 2023-03-03T06:16:46.581 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-03-03T06:16:46.799 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 117/134 2023-03-03T06:16:47.043 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-03-03T06:16:47.192 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-03-03T06:16:47.393 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-03-03T06:16:47.629 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-03-03T06:16:47.804 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 121/134 2023-03-03T06:16:47.976 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-03T06:16:48.442 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T06:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 115/134 2023-03-03T06:16:49.709 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-03-03T06:16:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 117/134 2023-03-03T06:16:50.090 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T06:16:50.236 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-03-03T06:16:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T06:16:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T06:16:50.395 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-03-03T06:16:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 124/134 2023-03-03T06:16:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-03-03T06:16:50.783 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 125/134 2023-03-03T06:16:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-03-03T06:16:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 121/134 2023-03-03T06:16:51.116 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 126/134 2023-03-03T06:16:51.245 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-03T06:16:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-03T06:16:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-03-03T06:16:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-03-03T06:16:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-03-03T06:16:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-03-03T06:16:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-03T06:16:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-03T06:16:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-03-03T06:16:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T06:16:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T06:16:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T06:16:52.756 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T06:16:52.924 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T06:16:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T06:16:53.215 INFO:teuthology.orchestra.run.smithi090.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-03T06:16:53.215 INFO:teuthology.orchestra.run.smithi090.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-03T06:16:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:16:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/134 2023-03-03T06:16:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/134 2023-03-03T06:16:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/134 2023-03-03T06:16:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/134 2023-03-03T06:16:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/134 2023-03-03T06:16:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T06:16:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T06:16:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T06:16:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/134 2023-03-03T06:16:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/134 2023-03-03T06:16:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/134 2023-03-03T06:16:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/134 2023-03-03T06:16:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/134 2023-03-03T06:16:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/134 2023-03-03T06:16:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/134 2023-03-03T06:16:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/134 2023-03-03T06:16:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/134 2023-03-03T06:16:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/134 2023-03-03T06:16:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/134 2023-03-03T06:16:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/134 2023-03-03T06:16:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/134 2023-03-03T06:16:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/134 2023-03-03T06:16:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/134 2023-03-03T06:16:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/134 2023-03-03T06:16:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-03T06:16:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-03T06:16:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-03T06:16:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-03T06:16:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-03T06:16:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-03T06:16:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-03T06:16:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-03T06:16:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-03T06:16:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-03T06:16:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-03T06:16:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-03T06:16:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-03T06:16:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-03T06:16:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-03T06:16:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-03T06:16:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-03T06:16:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-03T06:16:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-03T06:16:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-03T06:16:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-03T06:16:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-03T06:16:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-03T06:16:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-03T06:16:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-03T06:16:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-03T06:16:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-03-03T06:16:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-03-03T06:16:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-03-03T06:16:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-03-03T06:16:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-03-03T06:16:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-03-03T06:16:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-03-03T06:16:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-03-03T06:16:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-03-03T06:16:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-03-03T06:16:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-03-03T06:16:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-03-03T06:16:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-03-03T06:16:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-03-03T06:16:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-03-03T06:16:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-03-03T06:16:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-03-03T06:16:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-03-03T06:16:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-03-03T06:16:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-03-03T06:16:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-03-03T06:16:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-03-03T06:16:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-03-03T06:16:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-03-03T06:16:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-03-03T06:16:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-03-03T06:16:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-03-03T06:16:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-03-03T06:16:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-03-03T06:16:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-03-03T06:16:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-03-03T06:16:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-03-03T06:16:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-03-03T06:16:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-03-03T06:16:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-03-03T06:16:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-03-03T06:16:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 129/134 2023-03-03T06:16:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-03T06:16:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 131/134 2023-03-03T06:16:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T06:16:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-03T06:16:53.485 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T06:16:53.769 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T06:16:53.787 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T06:16:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 124/134 2023-03-03T06:16:54.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 125/134 2023-03-03T06:16:54.571 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 126/134 2023-03-03T06:16:54.928 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-03T06:16:55.151 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-03-03T06:16:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-03-03T06:16:55.460 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-03-03T06:16:55.472 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-03-03T06:16:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-03T06:16:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-03T06:16:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-03-03T06:16:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T06:16:56.046 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T06:16:56.047 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T06:16:56.123 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T06:16:56.123 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:59 ago on Fri 03 Mar 2023 06:14:57 AM UTC. 2023-03-03T06:16:56.266 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T06:16:56.500 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T06:16:56.772 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T06:16:56.772 INFO:teuthology.orchestra.run.smithi121.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-03T06:16:56.773 INFO:teuthology.orchestra.run.smithi121.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-03T06:16:56.773 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:16:56.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/134 2023-03-03T06:16:56.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/134 2023-03-03T06:16:56.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/134 2023-03-03T06:16:56.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/134 2023-03-03T06:16:56.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/134 2023-03-03T06:16:56.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T06:16:56.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T06:16:56.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T06:16:56.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/134 2023-03-03T06:16:56.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/134 2023-03-03T06:16:56.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/134 2023-03-03T06:16:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/134 2023-03-03T06:16:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/134 2023-03-03T06:16:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/134 2023-03-03T06:16:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/134 2023-03-03T06:16:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/134 2023-03-03T06:16:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/134 2023-03-03T06:16:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/134 2023-03-03T06:16:56.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/134 2023-03-03T06:16:56.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/134 2023-03-03T06:16:56.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/134 2023-03-03T06:16:56.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/134 2023-03-03T06:16:56.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/134 2023-03-03T06:16:56.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/134 2023-03-03T06:16:57.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/134 2023-03-03T06:16:57.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/134 2023-03-03T06:16:57.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/134 2023-03-03T06:16:57.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/134 2023-03-03T06:16:57.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/134 2023-03-03T06:16:57.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/134 2023-03-03T06:16:57.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/134 2023-03-03T06:16:57.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/134 2023-03-03T06:16:57.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/134 2023-03-03T06:16:57.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-03T06:16:57.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-03T06:16:57.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-03T06:16:57.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-03T06:16:57.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-03T06:16:57.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-03T06:16:57.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-03T06:16:57.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-03T06:16:57.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-03T06:16:57.004 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-03T06:16:57.005 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-03T06:16:57.005 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-03T06:16:57.005 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-03T06:16:57.005 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-03T06:16:57.005 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-03T06:16:57.005 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-03T06:16:57.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-03T06:16:57.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-03T06:16:57.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-03T06:16:57.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-03T06:16:57.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-03T06:16:57.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-03T06:16:57.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-03-03T06:16:57.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-03-03T06:16:57.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-03-03T06:16:57.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-03-03T06:16:57.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-03-03T06:16:57.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-03-03T06:16:57.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-03-03T06:16:57.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-03-03T06:16:57.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-03-03T06:16:57.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-03-03T06:16:57.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-03-03T06:16:57.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-03-03T06:16:57.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-03-03T06:16:57.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-03-03T06:16:57.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-03-03T06:16:57.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-03-03T06:16:57.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-03-03T06:16:57.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-03-03T06:16:57.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-03-03T06:16:57.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-03-03T06:16:57.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-03-03T06:16:57.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-03-03T06:16:57.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-03-03T06:16:57.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-03-03T06:16:57.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-03-03T06:16:57.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-03-03T06:16:57.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-03-03T06:16:57.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-03-03T06:16:57.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-03-03T06:16:57.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-03-03T06:16:57.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-03-03T06:16:57.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-03-03T06:16:57.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-03-03T06:16:57.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-03-03T06:16:57.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-03-03T06:16:57.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-03-03T06:16:57.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-03-03T06:16:57.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-03-03T06:16:57.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 129/134 2023-03-03T06:16:57.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-03T06:16:57.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 131/134 2023-03-03T06:16:57.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T06:16:57.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-03T06:16:58.662 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2023-03-03T06:16:58.897 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:16:58.897 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2023-03-03T06:16:58.897 INFO:teuthology.orchestra.run.smithi090.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-03T06:16:58.897 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.898 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.898 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2023-03-03T06:16:58.898 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-03T06:16:58.898 INFO:teuthology.orchestra.run.smithi090.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T06:16:58.899 INFO:teuthology.orchestra.run.smithi090.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T06:16:58.899 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.899 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.899 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.899 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.900 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T06:16:58.902 INFO:teuthology.orchestra.run.smithi090.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-03T06:16:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T06:16:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-3.el8.x86_64 2023-03-03T06:16:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T06:16:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-03T06:16:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T06:16:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-03T06:16:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-03T06:16:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T06:16:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T06:16:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-03T06:16:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T06:16:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-03T06:16:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-03T06:16:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-03T06:16:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-03T06:16:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T06:16:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T06:16:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T06:16:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T06:16:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T06:16:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T06:16:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T06:16:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T06:16:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T06:16:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T06:16:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T06:16:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T06:16:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T06:16:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T06:16:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T06:16:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T06:16:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T06:16:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T06:16:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-03T06:16:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T06:16:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T06:16:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T06:16:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:16:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T06:16:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T06:16:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T06:16:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T06:16:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T06:16:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-03T06:16:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:16:58.916 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2023-03-03T06:16:59.083 DEBUG:teuthology.parallel:result is None 2023-03-03T06:16:59.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T06:16:59.824 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:02 ago on Fri 03 Mar 2023 06:14:57 AM UTC. 2023-03-03T06:17:02.625 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-03-03T06:17:02.877 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:02.877 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2023-03-03T06:17:02.877 INFO:teuthology.orchestra.run.smithi121.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-03T06:17:02.878 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.878 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.878 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-03-03T06:17:02.878 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-03T06:17:02.878 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T06:17:02.879 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T06:17:02.879 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.879 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.879 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.879 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.879 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.879 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.880 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.880 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.880 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.880 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.880 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.880 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.881 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.881 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.881 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.881 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.881 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.881 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.881 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.882 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.882 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.882 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.882 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T06:17:02.882 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-03T06:17:02.882 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T06:17:02.883 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-03T06:17:02.883 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T06:17:02.883 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-3.el8.x86_64 2023-03-03T06:17:02.883 INFO:teuthology.orchestra.run.smithi121.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T06:17:02.883 INFO:teuthology.orchestra.run.smithi121.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-03T06:17:02.883 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T06:17:02.883 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.884 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.884 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.884 INFO:teuthology.orchestra.run.smithi121.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-03T06:17:02.884 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-03T06:17:02.884 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T06:17:02.884 INFO:teuthology.orchestra.run.smithi121.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T06:17:02.884 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-03T06:17:02.885 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T06:17:02.885 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.885 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.885 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T06:17:02.885 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.885 INFO:teuthology.orchestra.run.smithi121.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-03T06:17:02.886 INFO:teuthology.orchestra.run.smithi121.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T06:17:02.886 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T06:17:02.886 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T06:17:02.886 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-03T06:17:02.886 INFO:teuthology.orchestra.run.smithi121.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-03T06:17:02.886 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T06:17:02.886 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T06:17:02.887 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-03T06:17:02.887 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-03T06:17:02.887 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-03T06:17:02.887 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-03T06:17:02.887 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-03T06:17:02.887 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-03T06:17:02.887 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T06:17:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T06:17:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T06:17:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T06:17:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T06:17:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T06:17:02.890 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T06:17:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T06:17:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-03T06:17:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T06:17:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T06:17:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T06:17:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T06:17:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T06:17:02.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T06:17:02.893 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T06:17:02.893 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.893 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.893 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T06:17:02.893 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T06:17:02.893 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.893 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T06:17:02.894 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T06:17:02.895 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T06:17:02.895 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T06:17:02.895 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T06:17:02.895 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T06:17:02.895 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.895 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.895 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T06:17:02.896 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T06:17:02.896 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T06:17:02.896 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T06:17:02.896 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T06:17:02.896 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T06:17:02.897 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-03T06:17:02.897 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:02.897 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-03-03T06:17:03.056 DEBUG:teuthology.parallel:result is None 2023-03-03T06:17:03.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T06:17:03.195 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T06:17:03.224 INFO:teuthology.orchestra.run.smithi090.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T06:17:03.225 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T06:17:03.225 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T06:17:03.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T06:17:03.359 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T06:17:03.383 INFO:teuthology.orchestra.run.smithi121.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T06:17:03.383 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T06:17:03.383 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T06:17:03.384 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-03T06:17:03.384 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:03.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T06:17:03.430 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:03.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T06:17:03.469 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-03T06:17:03.470 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:03.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T06:17:03.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T06:17:03.567 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:03.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T06:17:03.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T06:17:03.660 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-03T06:17:03.660 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:03.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T06:17:03.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T06:17:03.755 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:03.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T06:17:03.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T06:17:03.847 INFO:teuthology.run_tasks:Running task ceph... 2023-03-03T06:17:03.954 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-03-03T06:17:03.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2023-03-03T06:17:03.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2023-03-03T06:17:03.983 INFO:tasks.ceph:Disabling ceph logrotate... 2023-03-03T06:17:03.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T06:17:04.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T06:17:04.051 INFO:tasks.ceph:Creating extra log directories... 2023-03-03T06:17:04.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T06:17:04.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T06:17:04.148 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-03-03T06:17:04.149 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-03-03T06:17:04.149 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-03-03T06:17:04.149 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:04.149 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T06:17:04.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:04.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:04.332 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:04.333 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T06:17:04.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:04.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:04.515 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-03-03T06:17:04.516 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-03-03T06:17:04.516 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140', 'branch': 'wip-vshankar-testing-20230303.023823', 'description': 'fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7192140', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20230303.023823'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20230303.023823', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20230303.023823', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi090.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', 'smithi121.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, {'ceph': None}, {'kclient': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6', 'timestamp': '2023-03-03_04:39:14', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.31660'} 2023-03-03T06:17:04.517 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T06:17:04.519 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T06:17:04.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T06:17:04.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T06:17:04.634 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:04.634 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T06:17:04.657 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T06:17:04.657 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2023-03-03T06:17:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T06:17:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:04.714 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 357024 Links: 1 2023-03-03T06:17:04.714 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:04.714 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:04.715 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-03-03 06:16:53.135614822 +0000 2023-03-03T06:17:04.715 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-03-03 06:14:03.769269199 +0000 2023-03-03T06:17:04.715 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-03-03 06:14:03.769269199 +0000 2023-03-03T06:17:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-03-03T06:17:04.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T06:17:04.779 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-03-03T06:17:04.779 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-03-03T06:17:04.779 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000247375 s, 2.1 MB/s 2023-03-03T06:17:04.780 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T06:17:04.837 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2023-03-03T06:17:04.894 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T06:17:04.894 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:04.894 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 360141 Links: 1 2023-03-03T06:17:04.894 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:04.895 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:04.895 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-03-03 06:16:53.135614822 +0000 2023-03-03T06:17:04.895 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-03-03 06:14:04.300260962 +0000 2023-03-03T06:17:04.895 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-03-03 06:14:04.300260962 +0000 2023-03-03T06:17:04.895 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-03-03T06:17:04.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T06:17:04.959 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-03-03T06:17:04.959 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-03-03T06:17:04.959 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000219864 s, 2.3 MB/s 2023-03-03T06:17:04.960 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T06:17:05.018 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2023-03-03T06:17:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T06:17:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:05.075 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 363227 Links: 1 2023-03-03T06:17:05.075 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:05.075 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:05.075 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-03-03 06:16:53.135614822 +0000 2023-03-03T06:17:05.076 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-03-03 06:14:04.811253036 +0000 2023-03-03T06:17:05.076 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-03-03 06:14:04.811253036 +0000 2023-03-03T06:17:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-03-03T06:17:05.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T06:17:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-03-03T06:17:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-03-03T06:17:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000220045 s, 2.3 MB/s 2023-03-03T06:17:05.142 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T06:17:05.200 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2023-03-03T06:17:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T06:17:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:05.258 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 358249 Links: 1 2023-03-03T06:17:05.258 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:05.258 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:05.258 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-03-03 06:16:53.135614822 +0000 2023-03-03T06:17:05.258 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-03-03 06:14:05.342244799 +0000 2023-03-03T06:17:05.258 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-03-03 06:14:05.342244799 +0000 2023-03-03T06:17:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-03-03T06:17:05.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T06:17:05.322 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-03-03T06:17:05.322 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-03-03T06:17:05.323 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000170401 s, 3.0 MB/s 2023-03-03T06:17:05.324 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T06:17:05.380 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:05.381 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:05.381 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T06:17:05.399 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T06:17:05.400 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2023-03-03T06:17:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T06:17:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:05.458 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 359903 Links: 1 2023-03-03T06:17:05.458 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:05.459 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:05.459 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-03 06:16:56.693680416 +0000 2023-03-03T06:17:05.459 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-03 06:14:09.443498241 +0000 2023-03-03T06:17:05.459 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-03 06:14:09.443498241 +0000 2023-03-03T06:17:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-03T06:17:05.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T06:17:05.526 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-03T06:17:05.526 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-03T06:17:05.526 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000249759 s, 2.0 MB/s 2023-03-03T06:17:05.527 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T06:17:05.586 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2023-03-03T06:17:05.645 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T06:17:05.645 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 363545 Links: 1 2023-03-03T06:17:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-03 06:16:56.693680416 +0000 2023-03-03T06:17:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-03 06:14:09.978491377 +0000 2023-03-03T06:17:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-03 06:14:09.978491377 +0000 2023-03-03T06:17:05.646 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-03T06:17:05.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T06:17:05.712 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-03T06:17:05.713 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-03T06:17:05.713 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.0002352 s, 2.2 MB/s 2023-03-03T06:17:05.714 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T06:17:05.773 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2023-03-03T06:17:05.831 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T06:17:05.832 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:05.832 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 358066 Links: 1 2023-03-03T06:17:05.832 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:05.832 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:05.832 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-03 06:16:56.693680416 +0000 2023-03-03T06:17:05.832 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-03 06:14:10.497484718 +0000 2023-03-03T06:17:05.832 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-03 06:14:10.497484718 +0000 2023-03-03T06:17:05.833 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-03T06:17:05.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T06:17:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-03T06:17:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-03T06:17:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000203782 s, 2.5 MB/s 2023-03-03T06:17:05.899 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T06:17:05.957 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2023-03-03T06:17:06.014 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T06:17:06.015 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T06:17:06.015 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 363625 Links: 1 2023-03-03T06:17:06.015 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:17:06.015 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T06:17:06.015 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-03 06:16:56.693680416 +0000 2023-03-03T06:17:06.015 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-03 06:14:11.050477622 +0000 2023-03-03T06:17:06.016 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-03 06:14:11.050477622 +0000 2023-03-03T06:17:06.016 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-03T06:17:06.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T06:17:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-03T06:17:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-03T06:17:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000249239 s, 2.1 MB/s 2023-03-03T06:17:06.081 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T06:17:06.139 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:06.139 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T06:17:06.140 INFO:tasks.ceph:Generating config... 2023-03-03T06:17:06.141 INFO:tasks.ceph:[client] client mount timeout = 600 2023-03-03T06:17:06.141 INFO:tasks.ceph:[client] debug client = 20 2023-03-03T06:17:06.142 INFO:tasks.ceph:[client] debug ms = 1 2023-03-03T06:17:06.142 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-03-03T06:17:06.142 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-03-03T06:17:06.142 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-03-03T06:17:06.142 INFO:tasks.ceph:[mds] debug mds = 20 2023-03-03T06:17:06.143 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-03-03T06:17:06.143 INFO:tasks.ceph:[mds] debug ms = 1 2023-03-03T06:17:06.143 INFO:tasks.ceph:[mds] mds debug frag = True 2023-03-03T06:17:06.143 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-03-03T06:17:06.143 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-03-03T06:17:06.144 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-03-03T06:17:06.144 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-03-03T06:17:06.144 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-03-03T06:17:06.144 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-03-03T06:17:06.144 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-03-03T06:17:06.144 INFO:tasks.ceph:[mgr] debug ms = 1 2023-03-03T06:17:06.145 INFO:tasks.ceph:[mon] debug mon = 20 2023-03-03T06:17:06.145 INFO:tasks.ceph:[mon] debug ms = 1 2023-03-03T06:17:06.145 INFO:tasks.ceph:[mon] debug paxos = 20 2023-03-03T06:17:06.146 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-03-03T06:17:06.146 INFO:tasks.ceph:[osd] bdev async discard = True 2023-03-03T06:17:06.146 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-03-03T06:17:06.146 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-03-03T06:17:06.146 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-03-03T06:17:06.147 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-03-03T06:17:06.147 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-03-03T06:17:06.147 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-03-03T06:17:06.147 INFO:tasks.ceph:[osd] debug ms = 1 2023-03-03T06:17:06.147 INFO:tasks.ceph:[osd] debug osd = 20 2023-03-03T06:17:06.148 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-03-03T06:17:06.148 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-03-03T06:17:06.148 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-03-03T06:17:06.148 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-03-03T06:17:06.148 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-03-03T06:17:06.149 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-03-03T06:17:06.149 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-03-03T06:17:06.149 INFO:tasks.ceph:Setting up mon.a... 2023-03-03T06:17:06.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-03-03T06:17:06.259 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.keyring 2023-03-03T06:17:06.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-03-03T06:17:06.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T06:17:06.375 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.90'), ('mon.b', '172.21.15.121'), ('mon.c', '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]')] 2023-03-03T06:17:06.377 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.90,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-03-03T06:17:06.378 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:06.379 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T06:17:06.486 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.90 --add b 172.21.15.121 --addv c '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:06.534 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:06.534 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: generated fsid 05bb16be-ef0c-4f48-a742-b70017592324 2023-03-03T06:17:06.534 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = pacific 2023-03-03T06:17:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 0 2023-03-03T06:17:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 05bb16be-ef0c-4f48-a742-b70017592324 2023-03-03T06:17:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:last_changed 2023-03-03T06:17:06.532932+0000 2023-03-03T06:17:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:created 2023-03-03T06:17:06.532932+0000 2023-03-03T06:17:06.536 INFO:teuthology.orchestra.run.smithi090.stdout:min_mon_release 16 (pacific) 2023-03-03T06:17:06.536 INFO:teuthology.orchestra.run.smithi090.stdout:election_strategy: 1 2023-03-03T06:17:06.536 INFO:teuthology.orchestra.run.smithi090.stdout:0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2023-03-03T06:17:06.537 INFO:teuthology.orchestra.run.smithi090.stdout:1: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2023-03-03T06:17:06.537 INFO:teuthology.orchestra.run.smithi090.stdout:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2023-03-03T06:17:06.537 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-03-03T06:17:06.539 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T06:17:06.556 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 05bb16be-ef0c-4f48-a742-b70017592324... 2023-03-03T06:17:06.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T06:17:06.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T06:17:06.641 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2023-03-03T06:17:06.641 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2023-03-03T06:17:06.641 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T06:17:06.642 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2023-03-03T06:17:06.642 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.642 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2023-03-03T06:17:06.642 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.642 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2023-03-03T06:17:06.642 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.643 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2023-03-03T06:17:06.643 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2023-03-03T06:17:06.643 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.643 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2023-03-03T06:17:06.643 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2023-03-03T06:17:06.643 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.643 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T06:17:06.644 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2023-03-03T06:17:06.644 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.644 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T06:17:06.644 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T06:17:06.644 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.644 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2023-03-03T06:17:06.644 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.645 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2023-03-03T06:17:06.645 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2023-03-03T06:17:06.645 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2023-03-03T06:17:06.645 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2023-03-03T06:17:06.645 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2023-03-03T06:17:06.645 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2023-03-03T06:17:06.645 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2023-03-03T06:17:06.646 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T06:17:06.646 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T06:17:06.646 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2023-03-03T06:17:06.646 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.646 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T06:17:06.646 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.646 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2023-03-03T06:17:06.647 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.647 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2023-03-03T06:17:06.647 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.647 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2023-03-03T06:17:06.647 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2023-03-03T06:17:06.647 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2023-03-03T06:17:06.648 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2023-03-03T06:17:06.648 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on skipped message = False 2023-03-03T06:17:06.648 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 05bb16be-ef0c-4f48-a742-b70017592324 2023-03-03T06:17:06.648 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.648 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2023-03-03T06:17:06.648 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2023-03-03T06:17:06.648 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.649 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2023-03-03T06:17:06.649 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2023-03-03T06:17:06.649 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.649 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2023-03-03T06:17:06.649 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2023-03-03T06:17:06.649 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.650 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2023-03-03T06:17:06.650 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2023-03-03T06:17:06.650 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2023-03-03T06:17:06.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.650 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2023-03-03T06:17:06.650 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2023-03-03T06:17:06.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.651 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2023-03-03T06:17:06.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.651 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2023-03-03T06:17:06.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.651 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2023-03-03T06:17:06.651 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.651 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2023-03-03T06:17:06.652 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2023-03-03T06:17:06.652 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.652 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2023-03-03T06:17:06.652 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2023-03-03T06:17:06.653 INFO:teuthology.orchestra.run.smithi090.stdout: bdev async discard = True 2023-03-03T06:17:06.653 INFO:teuthology.orchestra.run.smithi090.stdout: bdev enable discard = True 2023-03-03T06:17:06.653 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore allocator = bitmap 2023-03-03T06:17:06.653 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2023-03-03T06:17:06.653 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 1/20 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 1/20 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 4/10 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2023-03-03T06:17:06.654 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2023-03-03T06:17:06.655 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2023-03-03T06:17:06.655 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2023-03-03T06:17:06.655 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2023-03-03T06:17:06.655 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.656 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2023-03-03T06:17:06.656 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-03-03T06:17:06.656 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2023-03-03T06:17:06.656 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-03-03T06:17:06.656 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-03-03T06:17:06.656 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-03-03T06:17:06.656 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-03-03T06:17:06.657 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2023-03-03T06:17:06.657 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.657 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2023-03-03T06:17:06.657 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-03-03T06:17:06.657 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-03-03T06:17:06.657 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2023-03-03T06:17:06.658 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-03-03T06:17:06.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2023-03-03T06:17:06.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2023-03-03T06:17:06.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-03-03T06:17:06.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2023-03-03T06:17:06.658 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2023-03-03T06:17:06.659 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-03-03T06:17:06.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.659 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T06:17:06.659 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T06:17:06.659 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T06:17:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T06:17:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T06:17:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T06:17:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2023-03-03T06:17:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2023-03-03T06:17:06.661 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.661 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2023-03-03T06:17:06.661 INFO:teuthology.orchestra.run.smithi090.stdout: mon op complaint time = 120 2023-03-03T06:17:06.661 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:06.662 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2023-03-03T06:17:06.662 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2023-03-03T06:17:06.662 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2023-03-03T06:17:06.662 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2023-03-03T06:17:06.662 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T06:17:06.663 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T06:17:06.663 INFO:teuthology.orchestra.run.smithi090.stdout: client mount timeout = 600 2023-03-03T06:17:06.663 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2023-03-03T06:17:06.663 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-03-03T06:17:06.663 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2023-03-03T06:17:06.663 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2023-03-03T06:17:06.663 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2023-03-03T06:17:06.664 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2023-03-03T06:17:06.664 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2023-03-03T06:17:06.664 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2023-03-03T06:17:06.664 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2023-03-03T06:17:06.664 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2023-03-03T06:17:06.664 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2023-03-03T06:17:06.664 INFO:teuthology.orchestra.run.smithi090.stdout:[mds] 2023-03-03T06:17:06.665 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds = 20 2023-03-03T06:17:06.665 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds balancer = 20 2023-03-03T06:17:06.665 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-03-03T06:17:06.665 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug frag = True 2023-03-03T06:17:06.665 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug scatterstat = True 2023-03-03T06:17:06.665 INFO:teuthology.orchestra.run.smithi090.stdout: mds op complaint time = 180 2023-03-03T06:17:06.666 INFO:teuthology.orchestra.run.smithi090.stdout: mds verify scatter = True 2023-03-03T06:17:06.666 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2023-03-03T06:17:06.666 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2023-03-03T06:17:06.666 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2023-03-03T06:17:06.667 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2023-03-03T06:17:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2023-03-03T06:17:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T06:17:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2023-03-03T06:17:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.668 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2023-03-03T06:17:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2023-03-03T06:17:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2023-03-03T06:17:06.669 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2023-03-03T06:17:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2023-03-03T06:17:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2023-03-03T06:17:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T06:17:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2023-03-03T06:17:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T06:17:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T06:17:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2023-03-03T06:17:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2023-03-03T06:17:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2023-03-03T06:17:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2023-03-03T06:17:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2023-03-03T06:17:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2023-03-03T06:17:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2023-03-03T06:17:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2023-03-03T06:17:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T06:17:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T06:17:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2023-03-03T06:17:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T06:17:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2023-03-03T06:17:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2023-03-03T06:17:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2023-03-03T06:17:06.674 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2023-03-03T06:17:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2023-03-03T06:17:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.90,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2023-03-03T06:17:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on skipped message = False 2023-03-03T06:17:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 05bb16be-ef0c-4f48-a742-b70017592324 2023-03-03T06:17:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.676 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2023-03-03T06:17:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2023-03-03T06:17:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2023-03-03T06:17:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2023-03-03T06:17:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2023-03-03T06:17:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2023-03-03T06:17:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2023-03-03T06:17:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2023-03-03T06:17:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2023-03-03T06:17:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2023-03-03T06:17:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2023-03-03T06:17:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2023-03-03T06:17:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2023-03-03T06:17:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2023-03-03T06:17:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2023-03-03T06:17:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2023-03-03T06:17:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2023-03-03T06:17:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2023-03-03T06:17:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2023-03-03T06:17:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2023-03-03T06:17:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2023-03-03T06:17:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2023-03-03T06:17:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2023-03-03T06:17:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.683 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2023-03-03T06:17:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-03T06:17:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2023-03-03T06:17:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-03-03T06:17:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-03-03T06:17:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-03-03T06:17:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-03-03T06:17:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2023-03-03T06:17:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.684 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2023-03-03T06:17:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-03T06:17:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-03-03T06:17:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2023-03-03T06:17:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-03-03T06:17:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2023-03-03T06:17:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2023-03-03T06:17:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-03-03T06:17:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2023-03-03T06:17:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2023-03-03T06:17:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-03-03T06:17:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T06:17:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T06:17:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T06:17:06.686 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T06:17:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T06:17:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T06:17:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2023-03-03T06:17:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2023-03-03T06:17:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2023-03-03T06:17:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2023-03-03T06:17:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-03T06:17:06.689 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2023-03-03T06:17:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2023-03-03T06:17:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2023-03-03T06:17:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2023-03-03T06:17:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T06:17:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T06:17:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2023-03-03T06:17:06.691 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2023-03-03T06:17:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-03T06:17:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2023-03-03T06:17:06.692 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2023-03-03T06:17:06.692 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2023-03-03T06:17:06.692 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2023-03-03T06:17:06.692 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2023-03-03T06:17:06.692 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-03-03T06:17:06.693 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-03-03T06:17:06.693 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-03-03T06:17:06.693 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2023-03-03T06:17:06.693 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2023-03-03T06:17:06.693 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2023-03-03T06:17:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2023-03-03T06:17:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-03T06:17:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2023-03-03T06:17:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2023-03-03T06:17:06.694 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2023-03-03T06:17:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2023-03-03T06:17:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2023-03-03T06:17:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2023-03-03T06:17:06.695 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2023-03-03T06:17:06.696 INFO:tasks.ceph:Creating admin key on mon.a... 2023-03-03T06:17:06.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-03-03T06:17:06.763 INFO:tasks.ceph:Copying monmap to all nodes... 2023-03-03T06:17:06.764 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:06.764 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-03-03T06:17:06.781 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:06.781 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-03-03T06:17:06.837 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2023-03-03T06:17:06.838 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:06.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T06:17:06.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T06:17:06.913 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:06.913 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:06.970 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2023-03-03T06:17:06.970 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:06.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T06:17:06.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T06:17:07.011 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:07.011 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:07.069 INFO:tasks.ceph:Setting up mon nodes... 2023-03-03T06:17:07.069 INFO:tasks.ceph:Setting up mgr nodes... 2023-03-03T06:17:07.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T06:17:07.122 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T06:17:07.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T06:17:07.194 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T06:17:07.196 INFO:tasks.ceph:Setting up mds nodes... 2023-03-03T06:17:07.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T06:17:07.249 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T06:17:07.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-03-03T06:17:07.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T06:17:07.371 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T06:17:07.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-03-03T06:17:07.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T06:17:07.455 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T06:17:07.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-03-03T06:17:07.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T06:17:07.575 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T06:17:07.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-03-03T06:17:07.603 INFO:tasks.ceph_client:Setting up client nodes... 2023-03-03T06:17:07.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-03T06:17:07.647 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-03-03T06:17:07.660 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-03-03T06:17:07.661 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T06:17:07.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-03-03T06:17:07.686 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:07.687 INFO:tasks.ceph:role: osd.0 2023-03-03T06:17:07.687 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2023-03-03T06:17:07.687 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T06:17:07.760 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:07.760 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:07.760 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:07.760 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:07.760 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:07.760 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:07.761 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:07.761 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:07.761 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:07.761 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:08.181 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-03-03T06:17:08.182 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-03-03T06:17:08.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-03-03T06:17:08.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-03-03T06:17:08.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-03-03T06:17:08.445 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:08.445 INFO:tasks.ceph:role: osd.1 2023-03-03T06:17:08.445 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2023-03-03T06:17:08.446 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T06:17:08.518 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:08.519 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:08.519 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:08.519 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:08.519 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:08.519 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:08.519 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:08.520 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:08.520 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:08.520 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:08.949 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-03-03T06:17:08.950 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-03-03T06:17:08.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-03-03T06:17:09.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-03-03T06:17:09.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-03-03T06:17:09.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:09.174 INFO:tasks.ceph:role: osd.2 2023-03-03T06:17:09.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2023-03-03T06:17:09.174 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T06:17:09.248 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:09.249 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:09.249 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:09.249 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:09.249 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:09.249 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:09.250 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:09.250 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:09.250 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:09.250 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:09.708 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-03-03T06:17:09.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-03-03T06:17:09.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-03-03T06:17:09.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-03-03T06:17:09.916 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-03-03T06:17:09.984 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:09.984 INFO:tasks.ceph:role: osd.3 2023-03-03T06:17:09.985 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2023-03-03T06:17:09.985 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T06:17:10.060 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:10.060 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:10.061 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:10.061 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:10.061 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2023-03-03T06:17:10.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-03-03T06:17:10.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-03-03T06:17:10.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-03-03T06:17:10.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:10.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:10.866+0000 7ff34c118640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-03-03T06:17:10.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:10.866+0000 7ff34c118640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-03-03T06:17:10.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:10.866+0000 7ff34c118640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T06:17:10.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:10.866+0000 7ff34c118640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T06:17:10.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:10.866+0000 7ff34c118640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T06:17:10.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:10.867+0000 7ff34c118640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-03-03T06:17:13.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-03T06:17:13.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:13.872 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:13.870+0000 7f54fc0d8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-03-03T06:17:13.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:13.870+0000 7f54fc0d8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-03-03T06:17:13.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:13.870+0000 7f54fc0d8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T06:17:13.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:13.870+0000 7f54fc0d8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T06:17:13.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:13.870+0000 7f54fc0d8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T06:17:13.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:13.871+0000 7f54fc0d8640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-03-03T06:17:16.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-03T06:17:16.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:16.858+0000 7f633db06640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-03-03T06:17:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:16.858+0000 7f633db06640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-03-03T06:17:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:16.858+0000 7f633db06640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T06:17:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:16.858+0000 7f633db06640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T06:17:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:16.858+0000 7f633db06640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T06:17:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:16.859+0000 7f633db06640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-03-03T06:17:19.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-03T06:17:19.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:19.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:19.881+0000 7f617abdc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-03-03T06:17:19.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:19.881+0000 7f617abdc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-03-03T06:17:19.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:19.881+0000 7f617abdc640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T06:17:19.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:19.881+0000 7f617abdc640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T06:17:19.884 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:19.881+0000 7f617abdc640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T06:17:19.884 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:19.882+0000 7f617abdc640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-03-03T06:17:22.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-03T06:17:22.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-03-03T06:17:22.818 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:22.819 INFO:tasks.ceph:role: osd.4 2023-03-03T06:17:22.819 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-03T06:17:22.819 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T06:17:22.896 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:22.897 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:22.897 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:22.897 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:22.897 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:22.897 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:22.897 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:22.897 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:22.898 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:22.898 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:23.331 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-03T06:17:23.333 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-03T06:17:23.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-03-03T06:17:23.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-03-03T06:17:23.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-03-03T06:17:23.613 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:23.614 INFO:tasks.ceph:role: osd.5 2023-03-03T06:17:23.614 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-03T06:17:23.614 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T06:17:23.689 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:23.690 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:23.690 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:23.690 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:23.691 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:24.113 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-03T06:17:24.115 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-03T06:17:24.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-03-03T06:17:24.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-03-03T06:17:24.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-03-03T06:17:24.324 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:24.324 INFO:tasks.ceph:role: osd.6 2023-03-03T06:17:24.325 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-03T06:17:24.325 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T06:17:24.399 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:24.399 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:24.399 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:24.399 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:24.400 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:24.400 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:24.400 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:24.400 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:24.400 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:24.400 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:24.846 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-03T06:17:24.847 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-03T06:17:24.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-03-03T06:17:24.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-03-03T06:17:25.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-03-03T06:17:25.093 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T06:17:25.093 INFO:tasks.ceph:role: osd.7 2023-03-03T06:17:25.094 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-03T06:17:25.094 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T06:17:25.166 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T06:17:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T06:17:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T06:17:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T06:17:25.167 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T06:17:25.167 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-03T06:17:25.168 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T06:17:25.168 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T06:17:25.168 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T06:17:25.168 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T06:17:25.610 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-03T06:17:25.612 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-03T06:17:25.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-03-03T06:17:25.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-03-03T06:17:25.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:25.926 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:25.924+0000 7fe92bfdf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-03-03T06:17:25.926 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:25.924+0000 7fe92bfdf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-03-03T06:17:25.926 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:25.924+0000 7fe92bfdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T06:17:25.927 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:25.924+0000 7fe92bfdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T06:17:25.927 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:25.924+0000 7fe92bfdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T06:17:25.927 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:25.925+0000 7fe92bfdf640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-03-03T06:17:28.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-03T06:17:28.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:28.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:28.915+0000 7f68e6d3d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-03-03T06:17:28.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:28.915+0000 7f68e6d3d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-03-03T06:17:28.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:28.915+0000 7f68e6d3d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T06:17:28.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:28.915+0000 7f68e6d3d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T06:17:28.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:28.915+0000 7f68e6d3d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T06:17:28.918 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:28.915+0000 7f68e6d3d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-03-03T06:17:31.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-03-03T06:17:31.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:31.896 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:31.894+0000 7fee5104a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-03-03T06:17:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:31.894+0000 7fee5104a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-03-03T06:17:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:31.894+0000 7fee5104a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T06:17:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:31.894+0000 7fee5104a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T06:17:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:31.894+0000 7fee5104a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T06:17:31.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:31.895+0000 7fee5104a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-03-03T06:17:34.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:34.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:17:34.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-03-03T06:17:34.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:34.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:34.879+0000 7fd5e31e9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-03-03T06:17:34.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:34.879+0000 7fd5e31e9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-03-03T06:17:34.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:34.879+0000 7fd5e31e9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T06:17:34.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:34.879+0000 7fd5e31e9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T06:17:34.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:34.879+0000 7fd5e31e9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T06:17:34.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:34.880+0000 7fd5e31e9640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-03-03T06:17:37.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-03T06:17:37.791 INFO:tasks.ceph:Reading keys from all nodes... 2023-03-03T06:17:37.792 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:37.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-03-03T06:17:37.824 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:37.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-03-03T06:17:37.894 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:37.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-03-03T06:17:37.962 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:37.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-03-03T06:17:38.030 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:38.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-03-03T06:17:38.097 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:38.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-03-03T06:17:38.164 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:38.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-03-03T06:17:38.231 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:38.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-03-03T06:17:38.264 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:38.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-03-03T06:17:38.334 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:38.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-03-03T06:17:38.401 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:38.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-03-03T06:17:38.468 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:38.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-03-03T06:17:38.535 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:38.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-03-03T06:17:38.601 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:38.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-03-03T06:17:38.667 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:38.668 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-03-03T06:17:38.686 INFO:tasks.ceph:Adding keys to all mons... 2023-03-03T06:17:38.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T06:17:38.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T06:17:38.753 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.y] 2023-03-03T06:17:38.754 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDjkAFkF+dDBxAA1QdWYDYD5iEHQ6SLGqVusg== 2023-03-03T06:17:38.754 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2023-03-03T06:17:38.754 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDjkAFk4InZDhAACP0pM/T/3Y7MA3OdlY4ZaA== 2023-03-03T06:17:38.754 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2023-03-03T06:17:38.755 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDjkAFk7i4ZFhAAbioxMXTQ1uO5WfohMuyKLQ== 2023-03-03T06:17:38.755 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2023-03-03T06:17:38.755 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDmkAFkh8muMxAAPrXBpk6+3N/eySUY3YYHLQ== 2023-03-03T06:17:38.755 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2023-03-03T06:17:38.755 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDpkAFk0iHsMxAA1tu/xAO8wOIVx1pAtVY0Sw== 2023-03-03T06:17:38.755 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2023-03-03T06:17:38.756 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDskAFksKE1MxAAbezMM3aSR9SQIjnqTzZ9IQ== 2023-03-03T06:17:38.756 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2023-03-03T06:17:38.756 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDvkAFkYH2TNBAAYec5ObeUTtFSrk1wi1Wozw== 2023-03-03T06:17:38.756 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2023-03-03T06:17:38.757 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDjkAFktMiDCxAAW0AmzpfBSoozVt5UIDepwQ== 2023-03-03T06:17:38.757 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2023-03-03T06:17:38.757 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDjkAFkgnoQGxAABMQ0xN9PWytFD7BN9BxJ0g== 2023-03-03T06:17:38.757 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2023-03-03T06:17:38.758 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDjkAFkaBNHIhAAPuNxtBKcAgnU/MWQ/JYG7Q== 2023-03-03T06:17:38.758 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2023-03-03T06:17:38.758 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD1kAFkJIshNxAAKv3MSsPOpHtUHHIEUIVsLg== 2023-03-03T06:17:38.758 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2023-03-03T06:17:38.758 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD4kAFkSRyTNhAASd4eATypqZvfKKfcLpW5Dg== 2023-03-03T06:17:38.758 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.6] 2023-03-03T06:17:38.758 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD7kAFkUHNYNRAAOXOSjhGCaGO7SY4KEPBdPg== 2023-03-03T06:17:38.759 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.7] 2023-03-03T06:17:38.759 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQD+kAFkTtR3NBAA3bqvosGMbK2SEsmggz2lUA== 2023-03-03T06:17:38.759 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2023-03-03T06:17:38.759 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDjkAFkwCWQJhAABSbVrkyn43Bd+IU+t26Jcw== 2023-03-03T06:17:38.761 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2023-03-03T06:17:38.761 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDjkAFkF+dDBxAA1QdWYDYD5iEHQ6SLGqVusg== 2023-03-03T06:17:38.761 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-03-03T06:17:38.762 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDjkAFk4InZDhAACP0pM/T/3Y7MA3OdlY4ZaA== 2023-03-03T06:17:38.762 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-03-03T06:17:38.762 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDjkAFk7i4ZFhAAbioxMXTQ1uO5WfohMuyKLQ== 2023-03-03T06:17:38.762 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2023-03-03T06:17:38.762 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDmkAFkh8muMxAAPrXBpk6+3N/eySUY3YYHLQ== 2023-03-03T06:17:38.762 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2023-03-03T06:17:38.763 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDpkAFk0iHsMxAA1tu/xAO8wOIVx1pAtVY0Sw== 2023-03-03T06:17:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2023-03-03T06:17:38.763 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDskAFksKE1MxAAbezMM3aSR9SQIjnqTzZ9IQ== 2023-03-03T06:17:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2023-03-03T06:17:38.763 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDvkAFkYH2TNBAAYec5ObeUTtFSrk1wi1Wozw== 2023-03-03T06:17:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2023-03-03T06:17:38.763 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDjkAFktMiDCxAAW0AmzpfBSoozVt5UIDepwQ== 2023-03-03T06:17:38.764 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-03-03T06:17:38.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDjkAFkgnoQGxAABMQ0xN9PWytFD7BN9BxJ0g== 2023-03-03T06:17:38.764 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2023-03-03T06:17:38.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDjkAFkaBNHIhAAPuNxtBKcAgnU/MWQ/JYG7Q== 2023-03-03T06:17:38.764 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2023-03-03T06:17:38.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD1kAFkJIshNxAAKv3MSsPOpHtUHHIEUIVsLg== 2023-03-03T06:17:38.765 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2023-03-03T06:17:38.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD4kAFkSRyTNhAASd4eATypqZvfKKfcLpW5Dg== 2023-03-03T06:17:38.765 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2023-03-03T06:17:38.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD7kAFkUHNYNRAAOXOSjhGCaGO7SY4KEPBdPg== 2023-03-03T06:17:38.765 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2023-03-03T06:17:38.766 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQD+kAFkTtR3NBAA3bqvosGMbK2SEsmggz2lUA== 2023-03-03T06:17:38.766 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2023-03-03T06:17:38.766 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDjkAFkwCWQJhAABSbVrkyn43Bd+IU+t26Jcw== 2023-03-03T06:17:38.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T06:17:38.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T06:17:38.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:38.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:38.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:38.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:38.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:38.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T06:17:39.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T06:17:39.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:39.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:39.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:39.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T06:17:39.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T06:17:39.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T06:17:39.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T06:17:39.646 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-03-03T06:17:39.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-03-03T06:17:39.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T06:17:39.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-03-03T06:17:39.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-03-03T06:17:39.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T06:17:39.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-03-03T06:17:39.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-03-03T06:17:40.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T06:17:40.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-03-03T06:17:40.140 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:40.142 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T06:17:40.196 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-03-03T06:17:40.197 INFO:tasks.ceph.mon.a:Restarting daemon 2023-03-03T06:17:40.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-03-03T06:17:40.199 INFO:tasks.ceph.mon.a:Started 2023-03-03T06:17:40.199 INFO:tasks.ceph.mon.b:Restarting daemon 2023-03-03T06:17:40.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-03-03T06:17:40.239 INFO:tasks.ceph.mon.b:Started 2023-03-03T06:17:40.239 INFO:tasks.ceph.mon.c:Restarting daemon 2023-03-03T06:17:40.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-03-03T06:17:40.241 INFO:tasks.ceph.mon.c:Started 2023-03-03T06:17:40.241 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-03-03T06:17:40.242 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-03-03T06:17:40.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-03-03T06:17:40.244 INFO:tasks.ceph.mgr.y:Started 2023-03-03T06:17:40.244 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-03-03T06:17:40.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-03-03T06:17:40.246 INFO:tasks.ceph.mgr.x:Started 2023-03-03T06:17:40.247 INFO:tasks.ceph:Setting crush tunables to default 2023-03-03T06:17:40.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd crush tunables default 2023-03-03T06:17:40.369 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.366+0000 7f692c72b700 1 Processor -- start 2023-03-03T06:17:40.369 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.366+0000 7f692c72b700 1 -- start start 2023-03-03T06:17:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692c72b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692c72b700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692c72b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692406faf0 con 0x7f69240a5d10 2023-03-03T06:17:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692c72b700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69240b93c0 con 0x7f69240bf9e0 2023-03-03T06:17:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692c72b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692406f6e0 con 0x7f69240720c0 2023-03-03T06:17:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692acc8700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692acc8700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33244/0 (socket says 172.21.15.90:33244) 2023-03-03T06:17:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692acc8700 1 -- 172.21.15.90:0/1785862654 learned_addr learned my addr 172.21.15.90:0/1785862654 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692acc8700 1 -- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 msgr2=0x7f6924143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-03-03T06:17:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692acc8700 1 -- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 msgr2=0x7f6924143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T06:17:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692acc8700 1 --2- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T06:17:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692acc8700 1 --2- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-03T06:17:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692a4c7700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 msgr2=0x7f69240b9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-03-03T06:17:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692a4c7700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 msgr2=0x7f69240b9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T06:17:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T06:17:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.367+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-03T06:17:40.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692acc8700 1 --2- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692acc8700 1 -- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 msgr2=0x7f6924143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-03-03T06:17:40.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692acc8700 1 -- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 msgr2=0x7f6924143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T06:17:40.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692acc8700 1 --2- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T06:17:40.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692acc8700 1 --2- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-03T06:17:40.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692a4c7700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 msgr2=0x7f69240b9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-03-03T06:17:40.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692a4c7700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 msgr2=0x7f69240b9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T06:17:40.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T06:17:40.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.568+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-03T06:17:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.578+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633410094 0 0) 0x7f692406faf0 con 0x7f69240a5d10 2023-03-03T06:17:40.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.578+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69100035b0 con 0x7f69240a5d10 2023-03-03T06:17:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2659513489 0 0) 0x7f69100035b0 con 0x7f69240a5d10 2023-03-03T06:17:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 msgr2=0x7f69240b9b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T06:17:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f69240b9b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 msgr2=0x7f6924143030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T06:17:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 --2- 172.21.15.90:0/1785862654 >> v2:172.21.15.90:3300/0 conn(0x7f69240bf9e0 0x7f6924143030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692414da90 con 0x7f69240a5d10 2023-03-03T06:17:40.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f6920001670 con 0x7f69240a5d10 2023-03-03T06:17:40.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 >> v1:172.21.15.121:6789/0 conn(0x7f69240a5d10 legacy=0x7f69240bf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f6910003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f6910004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6910004d90 0x7f6910005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69240b93c0 con 0x7f6910004d90 2023-03-03T06:17:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692406f6e0 con 0x7f6910003a00 2023-03-03T06:17:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691000bbd0 con 0x7f69240720c0 2023-03-03T06:17:40.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f6910003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f6929cc6700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f6910004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692acc8700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6910004d90 0x7f6910005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692a4c7700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 msgr2=0x7f6910004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69240720c0 0x7f6910004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692a4c7700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6910004d90 msgr2=0x7f6910005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6910004d90 0x7f6910005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692a4c7700 1 -- 172.21.15.90:0/1785862654 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69100035b0 con 0x7f6910003a00 2023-03-03T06:17:40.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.579+0000 7f692acc8700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6910004d90 0x7f6910005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:40.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.580+0000 7f692a4c7700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f6910003e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f691400e900 tx=0x7f6910005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa5d18296248f470 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:40.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.581+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914015670 con 0x7f6910003a00 2023-03-03T06:17:40.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.581+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691400c050 con 0x7f6910003a00 2023-03-03T06:17:40.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.581+0000 7f69294c5700 1 -- 172.21.15.90:0/1785862654 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914015a30 con 0x7f6910003a00 2023-03-03T06:17:40.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.582+0000 7f692c72b700 1 -- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 msgr2=0x7f6910003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.582+0000 7f692c72b700 1 --2- 172.21.15.90:0/1785862654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f6910003e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f691400e900 tx=0x7f6910005880 comp rx=0 tx=0).stop 2023-03-03T06:17:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.582+0000 7f692acc8700 1 -- 172.21.15.90:0/1785862654 reap_dead start 2023-03-03T06:17:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.582+0000 7f692c72b700 1 -- 172.21.15.90:0/1785862654 shutdown_connections 2023-03-03T06:17:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.582+0000 7f692c72b700 1 -- 172.21.15.90:0/1785862654 >> 172.21.15.90:0/1785862654 conn(0x7f692406af80 msgr2=0x7f6924071740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.582+0000 7f692c72b700 1 -- 172.21.15.90:0/1785862654 shutdown_connections 2023-03-03T06:17:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.583+0000 7f692c72b700 1 -- 172.21.15.90:0/1785862654 wait complete. 2023-03-03T06:17:40.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.584+0000 7f692c72b700 1 Processor -- start 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.584+0000 7f692c72b700 1 -- start start 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f69240bdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f692414fca0 0x7f69241e3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6924146080 0x7f69241406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692414f2c0 con 0x7f6924146080 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692414e780 con 0x7f6910003a00 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69240be2a0 con 0x7f692414fca0 2023-03-03T06:17:40.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6924146080 0x7f69241406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692a4c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f69240bdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6924146080 0x7f69241406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33280/0 (socket says 172.21.15.90:33280) 2023-03-03T06:17:40.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 -- 172.21.15.90:0/518557698 learned_addr learned my addr 172.21.15.90:0/518557698 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:40.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f6929cc6700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f692414fca0 0x7f69241e3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:40.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 -- 172.21.15.90:0/518557698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f692414fca0 msgr2=0x7f69241e3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:40.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f692414fca0 0x7f69241e3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 -- 172.21.15.90:0/518557698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 msgr2=0x7f69240bdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f69240bdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 -- 172.21.15.90:0/518557698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69240b9610 con 0x7f6924146080 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692acc8700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6924146080 0x7f69241406f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6924073d70 tx=0x7f691c009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f691affd700 1 -- 172.21.15.90:0/518557698 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c003cf0 con 0x7f6924146080 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f691affd700 1 -- 172.21.15.90:0/518557698 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c00f050 con 0x7f6924146080 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f691affd700 1 -- 172.21.15.90:0/518557698 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c011900 con 0x7f6924146080 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692414dd30 con 0x7f6924146080 2023-03-03T06:17:40.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.585+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692406ebb0 con 0x7f6924146080 2023-03-03T06:17:40.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.586+0000 7f691affd700 1 -- 172.21.15.90:0/518557698 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f691c011a60 con 0x7f6924146080 2023-03-03T06:17:40.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.586+0000 7f691affd700 1 -- 172.21.15.90:0/518557698 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f691c004520 con 0x7f6924146080 2023-03-03T06:17:40.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.586+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692414c9b0 con 0x7f6924146080 2023-03-03T06:17:40.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.589+0000 7f691affd700 1 -- 172.21.15.90:0/518557698 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f691c019020 con 0x7f6924146080 2023-03-03T06:17:40.666 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.664+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f692414e8f0 con 0x7f6924146080 2023-03-03T06:17:40.673 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.671+0000 7f691affd700 1 -- 172.21.15.90:0/518557698 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f691c0098e0 con 0x7f6924146080 2023-03-03T06:17:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6924146080 msgr2=0x7f69241406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6924146080 0x7f69241406f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6924073d70 tx=0x7f691c009f70 comp rx=0 tx=0).stop 2023-03-03T06:17:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 shutdown_connections 2023-03-03T06:17:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6924146080 0x7f69241406f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6910003a00 0x7f69240bdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 --2- 172.21.15.90:0/518557698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f692414fca0 0x7f69241e3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 >> 172.21.15.90:0/518557698 conn(0x7f692406af80 msgr2=0x7f69240731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 shutdown_connections 2023-03-03T06:17:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:40.672+0000 7f692c72b700 1 -- 172.21.15.90:0/518557698 wait complete. 2023-03-03T06:17:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2023-03-03T06:17:40.686 INFO:tasks.ceph:check_enable_crimson: False 2023-03-03T06:17:40.687 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-03-03T06:17:40.687 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:40.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-03-03T06:17:40.698 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:40.697+0000 7f1e4d65f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T06:17:40.703 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:40.701+0000 7ff1b3fbf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T06:17:40.713 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:40.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-03-03T06:17:40.763 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:40.762+0000 7f1e4d65f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T06:17:40.774 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:40.771+0000 7ff1b3fbf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T06:17:40.779 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:40.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-03-03T06:17:40.836 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:40.835+0000 7f1e4d65f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T06:17:40.845 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:17:40.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-03-03T06:17:40.853 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:40.850+0000 7ff1b3fbf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T06:17:40.879 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:40.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-03-03T06:17:40.905 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:40.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-03-03T06:17:40.916 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:40.915+0000 7f1e4d65f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T06:17:40.936 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:40.934+0000 7ff1b3fbf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T06:17:40.943 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:40.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-03-03T06:17:41.009 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-03T06:17:41.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-03-03T06:17:41.065 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:41.064+0000 7f1e4d65f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T06:17:41.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c28f7dbf-67e2-451d-804b-7c735332e79d 0 2023-03-03T06:17:41.079 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:41.076+0000 7ff1b3fbf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T06:17:41.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.235+0000 7fc45a737700 1 Processor -- start 2023-03-03T06:17:41.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.235+0000 7fc45a737700 1 -- start start 2023-03-03T06:17:41.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc45a737700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc45414fe60 0x7fc4540c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc45a737700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc45a737700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc454146310 con 0x7fc4540c2860 2023-03-03T06:17:41.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc45a737700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45406d420 con 0x7fc45414fe60 2023-03-03T06:17:41.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc45a737700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45414cc50 con 0x7fc4540bd220 2023-03-03T06:17:41.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc458cd4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc453fff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc4540c2860 0x7fc45414f750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59298/0 (socket says 172.21.15.121:59298) 2023-03-03T06:17:41.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc453fff700 1 -- 172.21.15.121:0/845312564 learned_addr learned my addr 172.21.15.121:0/845312564 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-03T06:17:41.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc4537fe700 1 --2- 172.21.15.121:0/845312564 >> v2:172.21.15.90:3300/0 conn(0x7fc45414fe60 0x7fc4540c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.241 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc452ffd700 1 -- 172.21.15.121:0/845312564 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606324814 0 0) 0x7fc444001110 con 0x7fc4540c2860 2023-03-03T06:17:41.241 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc452ffd700 1 -- 172.21.15.121:0/845312564 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c0035b0 con 0x7fc4540c2860 2023-03-03T06:17:41.241 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc458cd4700 1 -- 172.21.15.121:0/845312564 >> v2:172.21.15.90:3300/0 conn(0x7fc45414fe60 msgr2=0x7fc4540c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:41.241 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc458cd4700 1 --2- 172.21.15.121:0/845312564 >> v2:172.21.15.90:3300/0 conn(0x7fc45414fe60 0x7fc4540c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.241 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc458cd4700 1 -- 172.21.15.121:0/845312564 >> v1:172.21.15.121:6789/0 conn(0x7fc4540c2860 legacy=0x7fc45414f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.236+0000 7fc458cd4700 1 -- 172.21.15.121:0/845312564 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540bd770 con 0x7fc4540bd220 2023-03-03T06:17:41.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc458cd4700 1 --2- 172.21.15.121:0/845312564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454156770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009d90 tx=0x7fc44c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2d8d19f46289da server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc452ffd700 1 -- 172.21.15.121:0/845312564 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00e070 con 0x7fc4540bd220 2023-03-03T06:17:41.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc452ffd700 1 -- 172.21.15.121:0/845312564 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c00bb20 con 0x7fc4540bd220 2023-03-03T06:17:41.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc452ffd700 1 -- 172.21.15.121:0/845312564 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c012430 con 0x7fc4540bd220 2023-03-03T06:17:41.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc45a737700 1 -- 172.21.15.121:0/845312564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 msgr2=0x7fc454156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc45a737700 1 --2- 172.21.15.121:0/845312564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454156770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009d90 tx=0x7fc44c00b330 comp rx=0 tx=0).stop 2023-03-03T06:17:41.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc45a737700 1 -- 172.21.15.121:0/845312564 shutdown_connections 2023-03-03T06:17:41.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc45a737700 1 --2- 172.21.15.121:0/845312564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454156770 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc45a737700 1 --2- 172.21.15.121:0/845312564 >> v2:172.21.15.90:3300/0 conn(0x7fc45414fe60 0x7fc4540c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.237+0000 7fc45a737700 1 -- 172.21.15.121:0/845312564 >> 172.21.15.121:0/845312564 conn(0x7fc45406c0d0 msgr2=0x7fc4540c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:41.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.238+0000 7fc45a737700 1 -- 172.21.15.121:0/845312564 shutdown_connections 2023-03-03T06:17:41.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.238+0000 7fc45a737700 1 -- 172.21.15.121:0/845312564 wait complete. 2023-03-03T06:17:41.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 Processor -- start 2023-03-03T06:17:41.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 -- start start 2023-03-03T06:17:41.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4540c2860 0x7fc4541e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc45414fe60 0x7fc4541420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc454148780 con 0x7fc45414fe60 2023-03-03T06:17:41.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc454145160 con 0x7fc4540c2860 2023-03-03T06:17:41.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc45a737700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4541e6260 con 0x7fc4540bd220 2023-03-03T06:17:41.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc4537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4540c2860 0x7fc4541e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.239+0000 7fc4537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4540c2860 0x7fc4541e5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46368/0 (socket says 172.21.15.121:46368) 2023-03-03T06:17:41.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc4537fe700 1 -- 172.21.15.121:0/3743273847 learned_addr learned my addr 172.21.15.121:0/3743273847 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:41.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc453fff700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc458cd4700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc45414fe60 0x7fc4541420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc4537fe700 1 -- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 msgr2=0x7fc454144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc4537fe700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454144c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc4537fe700 1 -- 172.21.15.121:0/3743273847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc45414fe60 msgr2=0x7fc4541420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc4537fe700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc45414fe60 0x7fc4541420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc4537fe700 1 -- 172.21.15.121:0/3743273847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c009a70 con 0x7fc4540c2860 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc4537fe700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4540c2860 0x7fc4541e5d20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc44800e870 tx=0x7fc44800eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc450ff9700 1 -- 172.21.15.121:0/3743273847 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448009590 con 0x7fc4540c2860 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc450ff9700 1 -- 172.21.15.121:0/3743273847 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448013680 con 0x7fc4540c2860 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4540bdd30 con 0x7fc4540c2860 2023-03-03T06:17:41.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc450ff9700 1 -- 172.21.15.121:0/3743273847 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4480139f0 con 0x7fc4540c2860 2023-03-03T06:17:41.249 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4540bda50 con 0x7fc4540c2860 2023-03-03T06:17:41.249 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc450ff9700 1 -- 172.21.15.121:0/3743273847 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fc448019030 con 0x7fc4540c2860 2023-03-03T06:17:41.249 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.240+0000 7fc450ff9700 1 -- 172.21.15.121:0/3743273847 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc4480064e0 con 0x7fc4540c2860 2023-03-03T06:17:41.249 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.241+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4540723b0 con 0x7fc4540c2860 2023-03-03T06:17:41.249 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.243+0000 7fc450ff9700 1 -- 172.21.15.121:0/3743273847 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fc448021060 con 0x7fc4540c2860 2023-03-03T06:17:41.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.326+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c28f7dbf-67e2-451d-804b-7c735332e79d", "id": 0} v 0) v1 -- 0x7fc4541488f0 con 0x7fc4540c2860 2023-03-03T06:17:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.332+0000 7fc450ff9700 1 -- 172.21.15.121:0/3743273847 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c28f7dbf-67e2-451d-804b-7c735332e79d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc448017070 con 0x7fc4540c2860 2023-03-03T06:17:41.334 INFO:teuthology.orchestra.run.smithi121.stdout:0 2023-03-03T06:17:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.333+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4540c2860 msgr2=0x7fc4541e5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.333+0000 7fc45a737700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4540c2860 0x7fc4541e5d20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc44800e870 tx=0x7fc44800eb80 comp rx=0 tx=0).stop 2023-03-03T06:17:41.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.334+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 shutdown_connections 2023-03-03T06:17:41.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.334+0000 7fc45a737700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc45414fe60 0x7fc4541420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.334+0000 7fc45a737700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4540bd220 0x7fc454144c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.334+0000 7fc45a737700 1 --2- 172.21.15.121:0/3743273847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4540c2860 0x7fc4541e5d20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.334+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 >> 172.21.15.121:0/3743273847 conn(0x7fc45406c0d0 msgr2=0x7fc45414f750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:41.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.334+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 shutdown_connections 2023-03-03T06:17:41.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.334+0000 7fc45a737700 1 -- 172.21.15.121:0/3743273847 wait complete. 2023-03-03T06:17:41.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7 1 2023-03-03T06:17:41.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.463+0000 7faaef433700 1 Processor -- start 2023-03-03T06:17:41.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.463+0000 7faaef433700 1 -- start start 2023-03-03T06:17:41.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faaef433700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7faae8150160 0x7faae8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faaef433700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae81505e0 con 0x7faae80bc4f0 2023-03-03T06:17:41.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faaef433700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae8066360 con 0x7faae814fcf0 2023-03-03T06:17:41.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faaef433700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae80bd920 con 0x7faae8150160 2023-03-03T06:17:41.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faaed9d0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7faae8150160 0x7faae8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faaec9ce700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7faae814fcf0 0x7faae81549e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59310/0 (socket says 172.21.15.121:59310) 2023-03-03T06:17:41.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faaec9ce700 1 -- 172.21.15.121:0/4152936741 learned_addr learned my addr 172.21.15.121:0/4152936741 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-03T06:17:41.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faadffff700 1 -- 172.21.15.121:0/4152936741 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800592894 0 0) 0x7faae8066360 con 0x7faae814fcf0 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.464+0000 7faadffff700 1 -- 172.21.15.121:0/4152936741 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad40035b0 con 0x7faae814fcf0 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faaed9d0700 1 -- 172.21.15.121:0/4152936741 >> v1:172.21.15.121:6789/0 conn(0x7faae814fcf0 legacy=0x7faae81549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faaed9d0700 1 -- 172.21.15.121:0/4152936741 >> v1:172.21.15.90:6789/0 conn(0x7faae80bc4f0 legacy=0x7faae80c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faaed9d0700 1 -- 172.21.15.121:0/4152936741 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae80bfeb0 con 0x7faae8150160 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faaed9d0700 1 --2- 172.21.15.121:0/4152936741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae8150160 0x7faae8158110 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faae0009d90 tx=0x7faae000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b0a6da0271e35e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faadffff700 1 -- 172.21.15.121:0/4152936741 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae000e070 con 0x7faae8150160 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faadffff700 1 -- 172.21.15.121:0/4152936741 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae000bb20 con 0x7faae8150160 2023-03-03T06:17:41.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faadffff700 1 -- 172.21.15.121:0/4152936741 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0012430 con 0x7faae8150160 2023-03-03T06:17:41.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faaef433700 1 -- 172.21.15.121:0/4152936741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae8150160 msgr2=0x7faae8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.465+0000 7faaef433700 1 --2- 172.21.15.121:0/4152936741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae8150160 0x7faae8158110 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faae0009d90 tx=0x7faae000b330 comp rx=0 tx=0).stop 2023-03-03T06:17:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.466+0000 7faaef433700 1 -- 172.21.15.121:0/4152936741 shutdown_connections 2023-03-03T06:17:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.466+0000 7faaef433700 1 --2- 172.21.15.121:0/4152936741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae8150160 0x7faae8158110 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.466+0000 7faaef433700 1 -- 172.21.15.121:0/4152936741 >> 172.21.15.121:0/4152936741 conn(0x7faae80c1900 msgr2=0x7faae80c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.466+0000 7faaef433700 1 -- 172.21.15.121:0/4152936741 shutdown_connections 2023-03-03T06:17:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.466+0000 7faaef433700 1 -- 172.21.15.121:0/4152936741 wait complete. 2023-03-03T06:17:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 Processor -- start 2023-03-03T06:17:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 -- start start 2023-03-03T06:17:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae80bc4f0 0x7faae80a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae814fcf0 0x7faae81e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faae8150160 0x7faae81e83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80bac30 con 0x7faae814fcf0 2023-03-03T06:17:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80a50a0 con 0x7faae80bc4f0 2023-03-03T06:17:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.467+0000 7faaef433700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae81e55e0 con 0x7faae8150160 2023-03-03T06:17:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faae8150160 0x7faae81e83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed1cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae80bc4f0 0x7faae80a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faae8150160 0x7faae81e83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53970/0 (socket says 172.21.15.121:53970) 2023-03-03T06:17:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 -- 172.21.15.121:0/464876464 learned_addr learned my addr 172.21.15.121:0/464876464 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed1cf700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae80bc4f0 0x7faae80a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46378/0 (socket says 172.21.15.121:46378) 2023-03-03T06:17:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaec9ce700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae814fcf0 0x7faae81e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 -- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae80bc4f0 msgr2=0x7faae80a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae80bc4f0 0x7faae80a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 -- 172.21.15.121:0/464876464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae814fcf0 msgr2=0x7faae81e50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae814fcf0 0x7faae81e50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 -- 172.21.15.121:0/464876464 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae0009a70 con 0x7faae8150160 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaed9d0700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faae8150160 0x7faae81e83d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faae806be20 tx=0x7faae0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaddffb700 1 -- 172.21.15.121:0/464876464 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0012430 con 0x7faae8150160 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaddffb700 1 -- 172.21.15.121:0/464876464 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae0020070 con 0x7faae8150160 2023-03-03T06:17:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaddffb700 1 -- 172.21.15.121:0/464876464 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0004280 con 0x7faae8150160 2023-03-03T06:17:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae806b620 con 0x7faae8150160 2023-03-03T06:17:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.468+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae80c0190 con 0x7faae8150160 2023-03-03T06:17:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.469+0000 7faaddffb700 1 -- 172.21.15.121:0/464876464 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7faae002e070 con 0x7faae8150160 2023-03-03T06:17:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.469+0000 7faaddffb700 1 -- 172.21.15.121:0/464876464 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7faae0022070 con 0x7faae8150160 2023-03-03T06:17:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.469+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae806b340 con 0x7faae8150160 2023-03-03T06:17:41.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.471+0000 7faaddffb700 1 -- 172.21.15.121:0/464876464 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7faae002c060 con 0x7faae8150160 2023-03-03T06:17:41.556 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.555+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7", "id": 1} v 0) v1 -- 0x7faae806f050 con 0x7faae8150160 2023-03-03T06:17:41.564 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.563+0000 7faaddffb700 1 -- 172.21.15.121:0/464876464 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7faae00048b0 con 0x7faae8150160 2023-03-03T06:17:41.565 INFO:teuthology.orchestra.run.smithi121.stdout:1 2023-03-03T06:17:41.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faae8150160 msgr2=0x7faae81e83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faae8150160 0x7faae81e83d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faae806be20 tx=0x7faae0004ad0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 shutdown_connections 2023-03-03T06:17:41.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faae8150160 0x7faae81e83d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faae80bc4f0 0x7faae80a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 --2- 172.21.15.121:0/464876464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae814fcf0 0x7faae81e50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 >> 172.21.15.121:0/464876464 conn(0x7faae80c1900 msgr2=0x7faae80a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:41.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 shutdown_connections 2023-03-03T06:17:41.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.564+0000 7faaef433700 1 -- 172.21.15.121:0/464876464 wait complete. 2023-03-03T06:17:41.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c3f69062-6ead-4f52-b01b-6213a6adfdb8 2 2023-03-03T06:17:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.696+0000 7f89e083c700 1 Processor -- start 2023-03-03T06:17:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.696+0000 7f89e083c700 1 -- start start 2023-03-03T06:17:41.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89e083c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc0c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89e083c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f89dc0bd220 0x7f89dc14f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89e083c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f89dc14fc90 0x7f89dc06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89e083c700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89dc146310 con 0x7f89dc14fc90 2023-03-03T06:17:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89e083c700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89dc1500f0 con 0x7f89dc0bd220 2023-03-03T06:17:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89e083c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc06d420 con 0x7f89dc0c2860 2023-03-03T06:17:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89da59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc0c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89d9d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f89dc0bd220 0x7f89dc14f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89da59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc0c2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53972/0 (socket says 172.21.15.121:53972) 2023-03-03T06:17:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.697+0000 7f89da59c700 1 -- 172.21.15.121:0/834189750 learned_addr learned my addr 172.21.15.121:0/834189750 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89dad9d700 1 --2- 172.21.15.121:0/834189750 >> v2:172.21.15.90:3300/0 conn(0x7f89dc14fc90 0x7f89dc06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89da59c700 1 -- 172.21.15.121:0/834189750 >> v2:172.21.15.121:3300/0 conn(0x7f89dc0bd220 msgr2=0x7f89dc14f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89da59c700 1 --2- 172.21.15.121:0/834189750 >> v2:172.21.15.121:3300/0 conn(0x7f89dc0bd220 0x7f89dc14f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89da59c700 1 -- 172.21.15.121:0/834189750 >> v2:172.21.15.90:3300/0 conn(0x7f89dc14fc90 msgr2=0x7f89dc06d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89da59c700 1 --2- 172.21.15.121:0/834189750 >> v2:172.21.15.90:3300/0 conn(0x7f89dc14fc90 0x7f89dc06d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89da59c700 1 -- 172.21.15.121:0/834189750 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc14cc50 con 0x7f89dc0c2860 2023-03-03T06:17:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89dad9d700 1 --2- 172.21.15.121:0/834189750 >> v2:172.21.15.90:3300/0 conn(0x7f89dc14fc90 0x7f89dc06d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:17:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89d9d9b700 1 --2- 172.21.15.121:0/834189750 >> v2:172.21.15.121:3300/0 conn(0x7f89dc0bd220 0x7f89dc14f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89da59c700 1 --2- 172.21.15.121:0/834189750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc0c2c30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0bfc30 tx=0x7f89cc009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12a9281074665b10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89d959a700 1 -- 172.21.15.121:0/834189750 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc011830 con 0x7f89dc0c2860 2023-03-03T06:17:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89d959a700 1 -- 172.21.15.121:0/834189750 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89cc011990 con 0x7f89dc0c2860 2023-03-03T06:17:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.698+0000 7f89d959a700 1 -- 172.21.15.121:0/834189750 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc011d20 con 0x7f89dc0c2860 2023-03-03T06:17:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 -- 172.21.15.121:0/834189750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 msgr2=0x7f89dc0c2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 --2- 172.21.15.121:0/834189750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc0c2c30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0bfc30 tx=0x7f89cc009bd0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 -- 172.21.15.121:0/834189750 shutdown_connections 2023-03-03T06:17:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 --2- 172.21.15.121:0/834189750 >> v2:172.21.15.90:3300/0 conn(0x7f89dc14fc90 0x7f89dc06d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 --2- 172.21.15.121:0/834189750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc0c2c30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 --2- 172.21.15.121:0/834189750 >> v2:172.21.15.121:3300/0 conn(0x7f89dc0bd220 0x7f89dc14f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 -- 172.21.15.121:0/834189750 >> 172.21.15.121:0/834189750 conn(0x7f89dc06c0d0 msgr2=0x7f89dc0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 -- 172.21.15.121:0/834189750 shutdown_connections 2023-03-03T06:17:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.699+0000 7f89e083c700 1 -- 172.21.15.121:0/834189750 wait complete. 2023-03-03T06:17:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.700+0000 7f89e083c700 1 Processor -- start 2023-03-03T06:17:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.700+0000 7f89e083c700 1 -- start start 2023-03-03T06:17:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89e083c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89dc0bd220 0x7f89dc0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89e083c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc14e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89e083c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89dc14fc90 0x7f89dc14e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89e083c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc1d7180 con 0x7f89dc14fc90 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89e083c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc1e56d0 con 0x7f89dc0bd220 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89e083c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc142190 con 0x7f89dc0c2860 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89dc0bd220 0x7f89dc0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89dc0bd220 0x7f89dc0c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46388/0 (socket says 172.21.15.121:46388) 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 -- 172.21.15.121:0/567990105 learned_addr learned my addr 172.21.15.121:0/567990105 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89d9d9b700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc14e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89dad9d700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89dc14fc90 0x7f89dc14e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 -- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 msgr2=0x7f89dc14e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc14e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 -- 172.21.15.121:0/567990105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89dc14fc90 msgr2=0x7f89dc14e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89dc14fc90 0x7f89dc14e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 -- 172.21.15.121:0/567990105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d00094a0 con 0x7f89dc0bd220 2023-03-03T06:17:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.701+0000 7f89da59c700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89dc0bd220 0x7f89dc0c12c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06d560 tx=0x7f89cc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.702+0000 7f89caffd700 1 -- 172.21.15.121:0/567990105 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc004450 con 0x7f89dc0bd220 2023-03-03T06:17:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.702+0000 7f89caffd700 1 -- 172.21.15.121:0/567990105 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89cc01e070 con 0x7f89dc0bd220 2023-03-03T06:17:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.702+0000 7f89caffd700 1 -- 172.21.15.121:0/567990105 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc0038e0 con 0x7f89dc0bd220 2023-03-03T06:17:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.702+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc009550 con 0x7f89dc0bd220 2023-03-03T06:17:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.702+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89dc0bda50 con 0x7f89dc0bd220 2023-03-03T06:17:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.702+0000 7f89caffd700 1 -- 172.21.15.121:0/567990105 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f89cc00f050 con 0x7f89dc0bd220 2023-03-03T06:17:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.702+0000 7f89caffd700 1 -- 172.21.15.121:0/567990105 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f89cc01a600 con 0x7f89dc0bd220 2023-03-03T06:17:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.703+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89dc0bdd30 con 0x7f89dc0bd220 2023-03-03T06:17:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.705+0000 7f89caffd700 1 -- 172.21.15.121:0/567990105 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f89cc032060 con 0x7f89dc0bd220 2023-03-03T06:17:41.785 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:41.782+0000 7ff1b3fbf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T06:17:41.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.788+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c3f69062-6ead-4f52-b01b-6213a6adfdb8", "id": 2} v 0) v1 -- 0x7f89dc0c1fb0 con 0x7f89dc0bd220 2023-03-03T06:17:41.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.796+0000 7f89caffd700 1 -- 172.21.15.121:0/567990105 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3f69062-6ead-4f52-b01b-6213a6adfdb8", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f89cc020070 con 0x7f89dc0bd220 2023-03-03T06:17:41.798 INFO:teuthology.orchestra.run.smithi121.stdout:2 2023-03-03T06:17:41.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89dc0bd220 msgr2=0x7f89dc0c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89dc0bd220 0x7f89dc0c12c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06d560 tx=0x7f89cc004750 comp rx=0 tx=0).stop 2023-03-03T06:17:41.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 shutdown_connections 2023-03-03T06:17:41.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89dc14fc90 0x7f89dc14e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89dc0bd220 0x7f89dc0c12c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 --2- 172.21.15.121:0/567990105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89dc0c2860 0x7f89dc14e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 >> 172.21.15.121:0/567990105 conn(0x7f89dc06c0d0 msgr2=0x7f89dc06a540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:41.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 shutdown_connections 2023-03-03T06:17:41.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.797+0000 7f89e083c700 1 -- 172.21.15.121:0/567990105 wait complete. 2023-03-03T06:17:41.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3 3 2023-03-03T06:17:41.835 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:41.833+0000 7f1e4d65f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T06:17:41.860 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:41.857+0000 7ff1b3fbf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T06:17:41.914 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:41.913+0000 7f1e4d65f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T06:17:41.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.931+0000 7f6f94751700 1 Processor -- start 2023-03-03T06:17:41.935 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.931+0000 7f6f94751700 1 -- start start 2023-03-03T06:17:41.937 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:41.931+0000 7ff1b3fbf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T06:17:41.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f94751700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0be4e0 0x7f6f8c151040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f94751700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6f8c072420 0x7f6f8c14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f94751700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c06ed70 con 0x7f6f8c0a4d00 2023-03-03T06:17:41.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f94751700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8c0713a0 con 0x7f6f8c072420 2023-03-03T06:17:41.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f94751700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c06d420 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f92cee700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6f8c072420 0x7f6f8c14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f91cec700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6f8c0a4d00 0x7f6f8c0c2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59336/0 (socket says 172.21.15.121:59336) 2023-03-03T06:17:41.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f91cec700 1 -- 172.21.15.121:0/4200330000 learned_addr learned my addr 172.21.15.121:0/4200330000 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-03T06:17:41.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.932+0000 7f6f924ed700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0be4e0 0x7f6f8c151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f92cee700 1 -- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0be4e0 msgr2=0x7f6f8c151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:41.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f92cee700 1 --2- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0be4e0 0x7f6f8c151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f92cee700 1 -- 172.21.15.121:0/4200330000 >> v1:172.21.15.121:6789/0 conn(0x7f6f8c0a4d00 legacy=0x7f6f8c0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f914eb700 1 -- 172.21.15.121:0/4200330000 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167888465 0 0) 0x7f6f8c06ed70 con 0x7f6f8c0a4d00 2023-03-03T06:17:41.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f92cee700 1 -- 172.21.15.121:0/4200330000 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c06ea90 con 0x7f6f8c072420 2023-03-03T06:17:41.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f924ed700 1 --2- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0be4e0 0x7f6f8c151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:41.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f92cee700 1 --2- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c072420 0x7f6f8c14ced0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009d90 tx=0x7f6f8400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=640bab60e4139a42 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f914eb700 1 -- 172.21.15.121:0/4200330000 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8400e070 con 0x7f6f8c072420 2023-03-03T06:17:41.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f914eb700 1 -- 172.21.15.121:0/4200330000 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8400bb20 con 0x7f6f8c072420 2023-03-03T06:17:41.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.933+0000 7f6f914eb700 1 -- 172.21.15.121:0/4200330000 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f84012430 con 0x7f6f8c072420 2023-03-03T06:17:41.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 -- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c072420 msgr2=0x7f6f8c14ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 --2- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c072420 0x7f6f8c14ced0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009d90 tx=0x7f6f8400b330 comp rx=0 tx=0).stop 2023-03-03T06:17:41.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 -- 172.21.15.121:0/4200330000 shutdown_connections 2023-03-03T06:17:41.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 --2- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c072420 0x7f6f8c14ced0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 --2- 172.21.15.121:0/4200330000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c0be4e0 0x7f6f8c151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 -- 172.21.15.121:0/4200330000 >> 172.21.15.121:0/4200330000 conn(0x7f6f8c06c0d0 msgr2=0x7f6f8c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:41.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 -- 172.21.15.121:0/4200330000 shutdown_connections 2023-03-03T06:17:41.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.934+0000 7f6f94751700 1 -- 172.21.15.121:0/4200330000 wait complete. 2023-03-03T06:17:41.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.935+0000 7f6f94751700 1 Processor -- start 2023-03-03T06:17:41.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.935+0000 7f6f94751700 1 -- start start 2023-03-03T06:17:41.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f94751700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f8c0a4d00 0x7f6f8c0c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f94751700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0be4e0 0x7f6f8c142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f94751700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c14fe00 0x7f6f8c0ba040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f94751700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c0bb6c0 con 0x7f6f8c0a4d00 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f94751700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c0b9700 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f94751700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c0b99e0 con 0x7f6f8c14fe00 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f91cec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0be4e0 0x7f6f8c142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f91cec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0be4e0 0x7f6f8c142a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46402/0 (socket says 172.21.15.121:46402) 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f91cec700 1 -- 172.21.15.121:0/1473660533 learned_addr learned my addr 172.21.15.121:0/1473660533 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:41.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f924ed700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f8c0a4d00 0x7f6f8c0c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f92cee700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c14fe00 0x7f6f8c0ba040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:41.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.936+0000 7f6f91cec700 1 -- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c14fe00 msgr2=0x7f6f8c0ba040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f91cec700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c14fe00 0x7f6f8c0ba040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f91cec700 1 -- 172.21.15.121:0/1473660533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f8c0a4d00 msgr2=0x7f6f8c0c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:41.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f91cec700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f8c0a4d00 0x7f6f8c0c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:41.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f91cec700 1 -- 172.21.15.121:0/1473660533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f880094a0 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f91cec700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0be4e0 0x7f6f8c142a90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c550 tx=0x7f6f7c00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:41.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f82ffd700 1 -- 172.21.15.121:0/1473660533 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c004c80 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f82ffd700 1 -- 172.21.15.121:0/1473660533 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c004e00 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f84009a70 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f82ffd700 1 -- 172.21.15.121:0/1473660533 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c007310 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c150840 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f82ffd700 1 -- 172.21.15.121:0/1473660533 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f6f7c01a020 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.937+0000 7f6f82ffd700 1 -- 172.21.15.121:0/1473660533 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f6f7c013850 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.938+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c150560 con 0x7f6f8c0be4e0 2023-03-03T06:17:41.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:41.940+0000 7f6f82ffd700 1 -- 172.21.15.121:0/1473660533 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f6f7c0075c0 con 0x7f6f8c0be4e0 2023-03-03T06:17:42.005 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:42.004+0000 7f1e4d65f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T06:17:42.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.023+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3", "id": 3} v 0) v1 -- 0x7f6f8c150b20 con 0x7f6f8c0be4e0 2023-03-03T06:17:42.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.027+0000 7f6f82ffd700 1 -- 172.21.15.121:0/1473660533 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6f7c01d520 con 0x7f6f8c0be4e0 2023-03-03T06:17:42.029 INFO:teuthology.orchestra.run.smithi121.stdout:3 2023-03-03T06:17:42.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.027+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0be4e0 msgr2=0x7f6f8c142a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.027+0000 7f6f94751700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0be4e0 0x7f6f8c142a90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c550 tx=0x7f6f7c00c860 comp rx=0 tx=0).stop 2023-03-03T06:17:42.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.028+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 shutdown_connections 2023-03-03T06:17:42.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.028+0000 7f6f94751700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f8c14fe00 0x7f6f8c0ba040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.028+0000 7f6f94751700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f8c0a4d00 0x7f6f8c0c29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.028+0000 7f6f94751700 1 --2- 172.21.15.121:0/1473660533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f8c0be4e0 0x7f6f8c142a90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.028+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 >> 172.21.15.121:0/1473660533 conn(0x7f6f8c06c0d0 msgr2=0x7f6f8c06a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.028+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 shutdown_connections 2023-03-03T06:17:42.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.028+0000 7f6f94751700 1 -- 172.21.15.121:0/1473660533 wait complete. 2023-03-03T06:17:42.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 68014d03-1e69-4652-a555-5a524933f4a1 4 2023-03-03T06:17:42.069 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:42.067+0000 7ff1b3fbf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T06:17:42.146 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:42.144+0000 7f1e4d65f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T06:17:42.203 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.201+0000 7f5d62a23700 1 Processor -- start 2023-03-03T06:17:42.203 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.201+0000 7f5d62a23700 1 -- start start 2023-03-03T06:17:42.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d62a23700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5d5c0bd670 0x7f5d5c0c09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d62a23700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5d5c06d9a0 0x7f5d5c06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d62a23700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c0c1060 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d62a23700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5c06d630 con 0x7f5d5c06d9a0 2023-03-03T06:17:42.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d62a23700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5c06cc90 con 0x7f5d5c0bd670 2023-03-03T06:17:42.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5d5c0bd670 0x7f5d5c0c09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5d5c0bd670 0x7f5d5c0c09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46406/0 (socket says 172.21.15.121:46406) 2023-03-03T06:17:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 -- 172.21.15.121:0/1888196176 learned_addr learned my addr 172.21.15.121:0/1888196176 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5b7fe700 1 --2- 172.21.15.121:0/1888196176 >> v2:172.21.15.90:3300/0 conn(0x7f5d5c06d9a0 0x7f5d5c06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 -- 172.21.15.121:0/1888196176 >> v2:172.21.15.90:3300/0 conn(0x7f5d5c06d9a0 msgr2=0x7f5d5c06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 --2- 172.21.15.121:0/1888196176 >> v2:172.21.15.90:3300/0 conn(0x7f5d5c06d9a0 0x7f5d5c06ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 -- 172.21.15.121:0/1888196176 >> v1:172.21.15.90:6789/0 conn(0x7f5d5c14f3f0 legacy=0x7f5d5c14f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 -- 172.21.15.121:0/1888196176 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c06cf70 con 0x7f5d5c0bd670 2023-03-03T06:17:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5b7fe700 1 --2- 172.21.15.121:0/1888196176 >> v2:172.21.15.90:3300/0 conn(0x7f5d5c06d9a0 0x7f5d5c06ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:17:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5bfff700 1 --2- 172.21.15.121:0/1888196176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c0bd670 0x7f5d5c0c09a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0be350 tx=0x7f5d4c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=731eb8297f2036d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5affd700 1 -- 172.21.15.121:0/1888196176 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c00b870 con 0x7f5d5c0bd670 2023-03-03T06:17:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5affd700 1 -- 172.21.15.121:0/1888196176 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c00b9d0 con 0x7f5d5c0bd670 2023-03-03T06:17:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.202+0000 7f5d5affd700 1 -- 172.21.15.121:0/1888196176 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c00bd60 con 0x7f5d5c0bd670 2023-03-03T06:17:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 -- 172.21.15.121:0/1888196176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c0bd670 msgr2=0x7f5d5c0c09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 --2- 172.21.15.121:0/1888196176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c0bd670 0x7f5d5c0c09a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0be350 tx=0x7f5d4c009bd0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 -- 172.21.15.121:0/1888196176 shutdown_connections 2023-03-03T06:17:42.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 --2- 172.21.15.121:0/1888196176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c0bd670 0x7f5d5c0c09a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 --2- 172.21.15.121:0/1888196176 >> v2:172.21.15.90:3300/0 conn(0x7f5d5c06d9a0 0x7f5d5c06ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 -- 172.21.15.121:0/1888196176 >> 172.21.15.121:0/1888196176 conn(0x7f5d5c0c1900 msgr2=0x7f5d5c0c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 -- 172.21.15.121:0/1888196176 shutdown_connections 2023-03-03T06:17:42.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.203+0000 7f5d62a23700 1 -- 172.21.15.121:0/1888196176 wait complete. 2023-03-03T06:17:42.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.205+0000 7f5d62a23700 1 Processor -- start 2023-03-03T06:17:42.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.205+0000 7f5d62a23700 1 -- start start 2023-03-03T06:17:42.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.205+0000 7f5d62a23700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d5c06d9a0 0x7f5d5c1e9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.205+0000 7f5d62a23700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d5c0bd670 0x7f5d5c1d9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d62a23700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c14f3f0 0x7f5d5c1d9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d62a23700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1d8b90 con 0x7f5d5c06d9a0 2023-03-03T06:17:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d62a23700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1d88b0 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d62a23700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1d9c50 con 0x7f5d5c0bd670 2023-03-03T06:17:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c14f3f0 0x7f5d5c1d9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c14f3f0 0x7f5d5c1d9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46416/0 (socket says 172.21.15.121:46416) 2023-03-03T06:17:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 -- 172.21.15.121:0/859589898 learned_addr learned my addr 172.21.15.121:0/859589898 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d5bfff700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d5c06d9a0 0x7f5d5c1e9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d5b7fe700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d5c0bd670 0x7f5d5c1d9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 -- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d5c0bd670 msgr2=0x7f5d5c1d9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d5c0bd670 0x7f5d5c1d9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 -- 172.21.15.121:0/859589898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d5c06d9a0 msgr2=0x7f5d5c1e9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d5c06d9a0 0x7f5d5c1e9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 -- 172.21.15.121:0/859589898 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c009550 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d60fc0700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c14f3f0 0x7f5d5c1d9560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400cb80 tx=0x7f5d5400ce90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d58ff9700 1 -- 172.21.15.121:0/859589898 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d54003c00 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d58ff9700 1 -- 172.21.15.121:0/859589898 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d54003d60 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c0728d0 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d58ff9700 1 -- 172.21.15.121:0/859589898 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5400aaa0 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c06d250 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.206+0000 7f5d58ff9700 1 -- 172.21.15.121:0/859589898 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f5d54010050 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.207+0000 7f5d58ff9700 1 -- 172.21.15.121:0/859589898 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f5d54005850 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.207+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c0bdf90 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.209+0000 7f5d58ff9700 1 -- 172.21.15.121:0/859589898 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f5d540050d0 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.294 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.293+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "68014d03-1e69-4652-a555-5a524933f4a1", "id": 4} v 0) v1 -- 0x7f5d5c1d8d40 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d58ff9700 1 -- 172.21.15.121:0/859589898 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "68014d03-1e69-4652-a555-5a524933f4a1", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5d5c1d8d40 con 0x7f5d5c14f3f0 2023-03-03T06:17:42.299 INFO:teuthology.orchestra.run.smithi121.stdout:4 2023-03-03T06:17:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c14f3f0 msgr2=0x7f5d5c1d9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d62a23700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c14f3f0 0x7f5d5c1d9560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400cb80 tx=0x7f5d5400ce90 comp rx=0 tx=0).stop 2023-03-03T06:17:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 shutdown_connections 2023-03-03T06:17:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d62a23700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d5c14f3f0 0x7f5d5c1d9560 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d62a23700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d5c06d9a0 0x7f5d5c1e9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d62a23700 1 --2- 172.21.15.121:0/859589898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d5c0bd670 0x7f5d5c1d9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.297+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 >> 172.21.15.121:0/859589898 conn(0x7f5d5c0c1900 msgr2=0x7f5d5c0bf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.298+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 shutdown_connections 2023-03-03T06:17:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.298+0000 7f5d62a23700 1 -- 172.21.15.121:0/859589898 wait complete. 2023-03-03T06:17:42.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 4c94930d-c84f-4f40-a615-d42ab7d8323a 5 2023-03-03T06:17:42.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.432+0000 7f53da338700 1 Processor -- start 2023-03-03T06:17:42.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.432+0000 7f53da338700 1 -- start start 2023-03-03T06:17:42.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53da338700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f53d40a6ef0 0x7f53d40a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53da338700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f53d40c2b30 0x7f53d4150720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53da338700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d4150c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53da338700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d40c0910 con 0x7f53d40a6ef0 2023-03-03T06:17:42.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53da338700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d4149770 con 0x7f53d40c2b30 2023-03-03T06:17:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53da338700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40c1a90 con 0x7f53d40c25e0 2023-03-03T06:17:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f53d40c2b30 0x7f53d4150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d88d5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d4150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f53d40c2b30 0x7f53d4150720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46420/0 (socket says 172.21.15.121:46420) 2023-03-03T06:17:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 -- 172.21.15.121:0/3556249096 learned_addr learned my addr 172.21.15.121:0/3556249096 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d3fff700 1 --2- 172.21.15.121:0/3556249096 >> v2:172.21.15.90:3300/0 conn(0x7f53d40a6ef0 0x7f53d40a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 -- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 msgr2=0x7f53d4150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 --2- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d4150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 -- 172.21.15.121:0/3556249096 >> v2:172.21.15.90:3300/0 conn(0x7f53d40a6ef0 msgr2=0x7f53d40a72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 --2- 172.21.15.121:0/3556249096 >> v2:172.21.15.90:3300/0 conn(0x7f53d40a6ef0 0x7f53d40a72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 -- 172.21.15.121:0/3556249096 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40c11d0 con 0x7f53d40c2b30 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d88d5700 1 --2- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d4150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.433+0000 7f53d37fe700 1 --2- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d4150720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f53d40c0ee0 tx=0x7f53c8009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa984cb73f4a8e13 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53d2ffd700 1 -- 172.21.15.121:0/3556249096 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8011830 con 0x7f53d40c2b30 2023-03-03T06:17:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53d2ffd700 1 -- 172.21.15.121:0/3556249096 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c8011990 con 0x7f53d40c2b30 2023-03-03T06:17:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53d2ffd700 1 -- 172.21.15.121:0/3556249096 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8011d20 con 0x7f53d40c2b30 2023-03-03T06:17:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 -- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 msgr2=0x7f53d4150720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 --2- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d4150720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f53d40c0ee0 tx=0x7f53c8009bd0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 -- 172.21.15.121:0/3556249096 shutdown_connections 2023-03-03T06:17:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 --2- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d4150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 --2- 172.21.15.121:0/3556249096 >> v2:172.21.15.90:3300/0 conn(0x7f53d40a6ef0 0x7f53d40a72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 --2- 172.21.15.121:0/3556249096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d4150720 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 -- 172.21.15.121:0/3556249096 >> 172.21.15.121:0/3556249096 conn(0x7f53d4071a60 msgr2=0x7f53d4071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 -- 172.21.15.121:0/3556249096 shutdown_connections 2023-03-03T06:17:42.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.434+0000 7f53da338700 1 -- 172.21.15.121:0/3556249096 wait complete. 2023-03-03T06:17:42.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.435+0000 7f53da338700 1 Processor -- start 2023-03-03T06:17:42.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53da338700 1 -- start start 2023-03-03T06:17:42.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53da338700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53d40a6ef0 0x7f53d40ba940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53da338700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d40bf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53da338700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d40b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53da338700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40a6d80 con 0x7f53d40a6ef0 2023-03-03T06:17:42.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53da338700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40b9290 con 0x7f53d40c2b30 2023-03-03T06:17:42.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53da338700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40b8f80 con 0x7f53d40c25e0 2023-03-03T06:17:42.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53d37fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d40bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53d88d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d40b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53d37fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d40bf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54014/0 (socket says 172.21.15.121:54014) 2023-03-03T06:17:42.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53d88d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d40b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46428/0 (socket says 172.21.15.121:46428) 2023-03-03T06:17:42.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53d37fe700 1 -- 172.21.15.121:0/1979146245 learned_addr learned my addr 172.21.15.121:0/1979146245 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:42.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.436+0000 7f53d3fff700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53d40a6ef0 0x7f53d40ba940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d37fe700 1 -- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 msgr2=0x7f53d40b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d37fe700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d40b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d37fe700 1 -- 172.21.15.121:0/1979146245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53d40a6ef0 msgr2=0x7f53d40ba940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d37fe700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53d40a6ef0 0x7f53d40ba940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d37fe700 1 -- 172.21.15.121:0/1979146245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc009a70 con 0x7f53d40c25e0 2023-03-03T06:17:42.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d88d5700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d40b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:42.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d37fe700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d40bf030 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d0ff9700 1 -- 172.21.15.121:0/1979146245 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8004450 con 0x7f53d40c25e0 2023-03-03T06:17:42.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d0ff9700 1 -- 172.21.15.121:0/1979146245 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c801e070 con 0x7f53d40c25e0 2023-03-03T06:17:42.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c8009550 con 0x7f53d40c25e0 2023-03-03T06:17:42.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d0ff9700 1 -- 172.21.15.121:0/1979146245 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c80038c0 con 0x7f53d40c25e0 2023-03-03T06:17:42.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d40c1440 con 0x7f53d40c25e0 2023-03-03T06:17:42.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d0ff9700 1 -- 172.21.15.121:0/1979146245 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f53c800f050 con 0x7f53d40c25e0 2023-03-03T06:17:42.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.437+0000 7f53d0ff9700 1 -- 172.21.15.121:0/1979146245 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f53c8003a20 con 0x7f53d40c25e0 2023-03-03T06:17:42.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.438+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d4150050 con 0x7f53d40c25e0 2023-03-03T06:17:42.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.440+0000 7f53d0ff9700 1 -- 172.21.15.121:0/1979146245 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f53c8020070 con 0x7f53d40c25e0 2023-03-03T06:17:42.454 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:42.451+0000 7ff1b3fbf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T06:17:42.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.522+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4c94930d-c84f-4f40-a615-d42ab7d8323a", "id": 5} v 0) v1 -- 0x7f53d40c1720 con 0x7f53d40c25e0 2023-03-03T06:17:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.528+0000 7f53d0ff9700 1 -- 172.21.15.121:0/1979146245 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4c94930d-c84f-4f40-a615-d42ab7d8323a", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f53c8015020 con 0x7f53d40c25e0 2023-03-03T06:17:42.529 INFO:teuthology.orchestra.run.smithi121.stdout:5 2023-03-03T06:17:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 msgr2=0x7f53d40bf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d40bf030 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c8004750 comp rx=0 tx=0).stop 2023-03-03T06:17:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 shutdown_connections 2023-03-03T06:17:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53d40c2b30 0x7f53d40b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53d40a6ef0 0x7f53d40ba940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 --2- 172.21.15.121:0/1979146245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53d40c25e0 0x7f53d40bf030 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 >> 172.21.15.121:0/1979146245 conn(0x7f53d4071a60 msgr2=0x7f53d41d8680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 shutdown_connections 2023-03-03T06:17:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.529+0000 7f53da338700 1 -- 172.21.15.121:0/1979146245 wait complete. 2023-03-03T06:17:42.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 3c0c3008-2ed7-4db3-9b17-3226864daff8 6 2023-03-03T06:17:42.552 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:42.551+0000 7f1e4d65f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T06:17:42.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.662+0000 7f824fd4a700 1 Processor -- start 2023-03-03T06:17:42.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824fd4a700 1 -- start start 2023-03-03T06:17:42.666 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824fd4a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f824814fcf0 0x7f8248150160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824fd4a700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8248066360 con 0x7f82481549e0 2023-03-03T06:17:42.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824fd4a700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82480bfeb0 con 0x7f82480bc4f0 2023-03-03T06:17:42.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824fd4a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82480bd920 con 0x7f824814fcf0 2023-03-03T06:17:42.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824d2e5700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f824814fcf0 0x7f8248150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824dae6700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f82480bc4f0 0x7f82480c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59368/0 (socket says 172.21.15.121:59368) 2023-03-03T06:17:42.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824d2e5700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f824814fcf0 0x7f8248150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46432/0 (socket says 172.21.15.121:46432) 2023-03-03T06:17:42.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.663+0000 7f824dae6700 1 -- 172.21.15.121:0/1031798792 learned_addr learned my addr 172.21.15.121:0/1031798792 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-03T06:17:42.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.664+0000 7f824d2e5700 1 -- 172.21.15.121:0/1031798792 >> v1:172.21.15.121:6789/0 conn(0x7f82480bc4f0 legacy=0x7f82480c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.664+0000 7f824d2e5700 1 -- 172.21.15.121:0/1031798792 >> v1:172.21.15.90:6789/0 conn(0x7f82481549e0 legacy=0x7f8248156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.664+0000 7f824d2e5700 1 -- 172.21.15.121:0/1031798792 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480c0190 con 0x7f824814fcf0 2023-03-03T06:17:42.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.664+0000 7f824d2e5700 1 --2- 172.21.15.121:0/1031798792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f8248150160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8238009870 tx=0x7f8238009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=798b2a3d58d9734 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.664+0000 7f824cae4700 1 -- 172.21.15.121:0/1031798792 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238011830 con 0x7f824814fcf0 2023-03-03T06:17:42.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.664+0000 7f824cae4700 1 -- 172.21.15.121:0/1031798792 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238011990 con 0x7f824814fcf0 2023-03-03T06:17:42.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.664+0000 7f824cae4700 1 -- 172.21.15.121:0/1031798792 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238011d20 con 0x7f824814fcf0 2023-03-03T06:17:42.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.665+0000 7f824fd4a700 1 -- 172.21.15.121:0/1031798792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 msgr2=0x7f8248150160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.665+0000 7f824fd4a700 1 --2- 172.21.15.121:0/1031798792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f8248150160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8238009870 tx=0x7f8238009b80 comp rx=0 tx=0).stop 2023-03-03T06:17:42.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.665+0000 7f824fd4a700 1 -- 172.21.15.121:0/1031798792 shutdown_connections 2023-03-03T06:17:42.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.665+0000 7f824fd4a700 1 --2- 172.21.15.121:0/1031798792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f8248150160 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.665+0000 7f824fd4a700 1 -- 172.21.15.121:0/1031798792 >> 172.21.15.121:0/1031798792 conn(0x7f82480c1900 msgr2=0x7f82480c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.665+0000 7f824fd4a700 1 -- 172.21.15.121:0/1031798792 shutdown_connections 2023-03-03T06:17:42.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.665+0000 7f824fd4a700 1 -- 172.21.15.121:0/1031798792 wait complete. 2023-03-03T06:17:42.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 Processor -- start 2023-03-03T06:17:42.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 -- start start 2023-03-03T06:17:42.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82480bc4f0 0x7f82481446f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f82481df400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82481549e0 0x7f82481df940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82481e3260 con 0x7f82481549e0 2023-03-03T06:17:42.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82481e3540 con 0x7f824814fcf0 2023-03-03T06:17:42.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824fd4a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824814f070 con 0x7f82480bc4f0 2023-03-03T06:17:42.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824d2e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f82481df400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.666+0000 7f824dae6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82480bc4f0 0x7f82481446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f82481df400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46438/0 (socket says 172.21.15.121:46438) 2023-03-03T06:17:42.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824dae6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82480bc4f0 0x7f82481446f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54024/0 (socket says 172.21.15.121:54024) 2023-03-03T06:17:42.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 -- 172.21.15.121:0/195703539 learned_addr learned my addr 172.21.15.121:0/195703539 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:42.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824e2e7700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82481549e0 0x7f82481df940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 -- 172.21.15.121:0/195703539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82480bc4f0 msgr2=0x7f82481446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82480bc4f0 0x7f82481446f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 -- 172.21.15.121:0/195703539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82481549e0 msgr2=0x7f82481df940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82481549e0 0x7f82481df940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 -- 172.21.15.121:0/195703539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82380094a0 con 0x7f824814fcf0 2023-03-03T06:17:42.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824d2e5700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f82481df400 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f82380090b0 tx=0x7f8238004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f823e7fc700 1 -- 172.21.15.121:0/195703539 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82380036a0 con 0x7f824814fcf0 2023-03-03T06:17:42.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f823e7fc700 1 -- 172.21.15.121:0/195703539 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823801f070 con 0x7f824814fcf0 2023-03-03T06:17:42.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f823e7fc700 1 -- 172.21.15.121:0/195703539 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238003a60 con 0x7f824814fcf0 2023-03-03T06:17:42.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824fd4a700 1 -- 172.21.15.121:0/195703539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824806b340 con 0x7f824814fcf0 2023-03-03T06:17:42.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f824fd4a700 1 -- 172.21.15.121:0/195703539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824806b620 con 0x7f824814fcf0 2023-03-03T06:17:42.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f823e7fc700 1 -- 172.21.15.121:0/195703539 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f823800f050 con 0x7f824814fcf0 2023-03-03T06:17:42.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.667+0000 7f823e7fc700 1 -- 172.21.15.121:0/195703539 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f8238021070 con 0x7f824814fcf0 2023-03-03T06:17:42.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.669+0000 7f822bfff700 1 -- 172.21.15.121:0/195703539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480bdaf0 con 0x7f824814fcf0 2023-03-03T06:17:42.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.671+0000 7f823e7fc700 1 -- 172.21.15.121:0/195703539 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f823802b020 con 0x7f824814fcf0 2023-03-03T06:17:42.770 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.768+0000 7f822bfff700 1 -- 172.21.15.121:0/195703539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3c0c3008-2ed7-4db3-9b17-3226864daff8", "id": 6} v 0) v1 -- 0x7f82481e36b0 con 0x7f824814fcf0 2023-03-03T06:17:42.776 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.774+0000 7f823e7fc700 1 -- 172.21.15.121:0/195703539 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3c0c3008-2ed7-4db3-9b17-3226864daff8", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8238015020 con 0x7f824814fcf0 2023-03-03T06:17:42.776 INFO:teuthology.orchestra.run.smithi121.stdout:6 2023-03-03T06:17:42.777 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.775+0000 7f822bfff700 1 -- 172.21.15.121:0/195703539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 msgr2=0x7f82481df400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.777 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.775+0000 7f822bfff700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f82481df400 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f82380090b0 tx=0x7f8238004750 comp rx=0 tx=0).stop 2023-03-03T06:17:42.777 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.775+0000 7f822bfff700 1 -- 172.21.15.121:0/195703539 shutdown_connections 2023-03-03T06:17:42.777 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.775+0000 7f822bfff700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82481549e0 0x7f82481df940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.778 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.775+0000 7f822bfff700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82480bc4f0 0x7f82481446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.778 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.775+0000 7f822bfff700 1 --2- 172.21.15.121:0/195703539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f824814fcf0 0x7f82481df400 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.778 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.775+0000 7f822bfff700 1 -- 172.21.15.121:0/195703539 >> 172.21.15.121:0/195703539 conn(0x7f82480c1900 msgr2=0x7f824806fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.778 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.776+0000 7f822bfff700 1 -- 172.21.15.121:0/195703539 shutdown_connections 2023-03-03T06:17:42.778 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.776+0000 7f822bfff700 1 -- 172.21.15.121:0/195703539 wait complete. 2023-03-03T06:17:42.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new aa96b041-9dcd-4888-9b17-4cdaaa907e4a 7 2023-03-03T06:17:42.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.952+0000 7f7f71ea4700 1 Processor -- start 2023-03-03T06:17:42.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.952+0000 7f7f71ea4700 1 -- start start 2023-03-03T06:17:42.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f71ea4700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7f6c0c2860 0x7f7f6c0c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f71ea4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f71ea4700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c146310 con 0x7f7f6c0bd220 2023-03-03T06:17:42.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f71ea4700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6c06d420 con 0x7f7f6c0c2860 2023-03-03T06:17:42.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f71ea4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c14cc50 con 0x7f7f6c14fe60 2023-03-03T06:17:42.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54034/0 (socket says 172.21.15.121:54034) 2023-03-03T06:17:42.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 -- 172.21.15.121:0/270488877 learned_addr learned my addr 172.21.15.121:0/270488877 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:42.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6b7fe700 1 --2- 172.21.15.121:0/270488877 >> v2:172.21.15.90:3300/0 conn(0x7f7f6c0c2860 0x7f7f6c0c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 -- 172.21.15.121:0/270488877 >> v2:172.21.15.90:3300/0 conn(0x7f7f6c0c2860 msgr2=0x7f7f6c0c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:42.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 --2- 172.21.15.121:0/270488877 >> v2:172.21.15.90:3300/0 conn(0x7f7f6c0c2860 0x7f7f6c0c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 -- 172.21.15.121:0/270488877 >> v1:172.21.15.90:6789/0 conn(0x7f7f6c0bd220 legacy=0x7f7f6c14f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 -- 172.21.15.121:0/270488877 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c0bd770 con 0x7f7f6c14fe60 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.953+0000 7f7f6bfff700 1 --2- 172.21.15.121:0/270488877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c156770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f640098f0 tx=0x7f7f64009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4746861b3103e483 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f6a7fc700 1 -- 172.21.15.121:0/270488877 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64011830 con 0x7f7f6c14fe60 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f6a7fc700 1 -- 172.21.15.121:0/270488877 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f64011990 con 0x7f7f6c14fe60 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f6a7fc700 1 -- 172.21.15.121:0/270488877 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64011d20 con 0x7f7f6c14fe60 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 -- 172.21.15.121:0/270488877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 msgr2=0x7f7f6c156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 --2- 172.21.15.121:0/270488877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c156770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f640098f0 tx=0x7f7f64009c00 comp rx=0 tx=0).stop 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 -- 172.21.15.121:0/270488877 shutdown_connections 2023-03-03T06:17:42.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 --2- 172.21.15.121:0/270488877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c156770 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 --2- 172.21.15.121:0/270488877 >> v2:172.21.15.90:3300/0 conn(0x7f7f6c0c2860 0x7f7f6c0c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 -- 172.21.15.121:0/270488877 >> 172.21.15.121:0/270488877 conn(0x7f7f6c06c0d0 msgr2=0x7f7f6c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 -- 172.21.15.121:0/270488877 shutdown_connections 2023-03-03T06:17:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.954+0000 7f7f71ea4700 1 -- 172.21.15.121:0/270488877 wait complete. 2023-03-03T06:17:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.955+0000 7f7f71ea4700 1 Processor -- start 2023-03-03T06:17:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f71ea4700 1 -- start start 2023-03-03T06:17:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f71ea4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f6c0bd220 0x7f7f6c0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f71ea4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f6c0c2860 0x7f7f6c14b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f71ea4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c14ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f71ea4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0bbe10 con 0x7f7f6c0c2860 2023-03-03T06:17:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f71ea4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0a64f0 con 0x7f7f6c0bd220 2023-03-03T06:17:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f71ea4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c149420 con 0x7f7f6c14fe60 2023-03-03T06:17:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c14ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f6b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f6c0bd220 0x7f7f6c0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c14ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54040/0 (socket says 172.21.15.121:54040) 2023-03-03T06:17:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f6bfff700 1 -- 172.21.15.121:0/3268740224 learned_addr learned my addr 172.21.15.121:0/3268740224 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-03T06:17:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f6affd700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f6c0c2860 0x7f7f6c14b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.956+0000 7f7f6bfff700 1 -- 172.21.15.121:0/3268740224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f6c0bd220 msgr2=0x7f7f6c0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f6bfff700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f6c0bd220 0x7f7f6c0a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f6bfff700 1 -- 172.21.15.121:0/3268740224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f6c0c2860 msgr2=0x7f7f6c14b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f6bfff700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f6c0c2860 0x7f7f6c14b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f6bfff700 1 -- 172.21.15.121:0/3268740224 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64009550 con 0x7f7f6c14fe60 2023-03-03T06:17:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f6bfff700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c14ba80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7f64009c40 tx=0x7f7f64004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f53fff700 1 -- 172.21.15.121:0/3268740224 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64004510 con 0x7f7f6c14fe60 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f53fff700 1 -- 172.21.15.121:0/3268740224 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6401e070 con 0x7f7f6c14fe60 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c0bdd30 con 0x7f7f6c14fe60 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f53fff700 1 -- 172.21.15.121:0/3268740224 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f640038e0 con 0x7f7f6c14fe60 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c0bda50 con 0x7f7f6c14fe60 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f53fff700 1 -- 172.21.15.121:0/3268740224 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f7f6400f050 con 0x7f7f6c14fe60 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.957+0000 7f7f53fff700 1 -- 172.21.15.121:0/3268740224 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7f6401abc0 con 0x7f7f6c14fe60 2023-03-03T06:17:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.958+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c0723b0 con 0x7f7f6c14fe60 2023-03-03T06:17:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:42.960+0000 7f7f53fff700 1 -- 172.21.15.121:0/3268740224 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7f64032060 con 0x7f7f6c14fe60 2023-03-03T06:17:43.015 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:43.013+0000 7ff1b3fbf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T06:17:43.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.076+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "aa96b041-9dcd-4888-9b17-4cdaaa907e4a", "id": 7} v 0) v1 -- 0x7f7f6c04e750 con 0x7f7f6c14fe60 2023-03-03T06:17:43.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.082+0000 7f7f53fff700 1 -- 172.21.15.121:0/3268740224 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aa96b041-9dcd-4888-9b17-4cdaaa907e4a", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7f64020070 con 0x7f7f6c14fe60 2023-03-03T06:17:43.083 INFO:teuthology.orchestra.run.smithi121.stdout:7 2023-03-03T06:17:43.084 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:43.081+0000 7ff1b3fbf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T06:17:43.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 msgr2=0x7f7f6c14ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:43.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c14ba80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7f64009c40 tx=0x7f7f64004750 comp rx=0 tx=0).stop 2023-03-03T06:17:43.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 shutdown_connections 2023-03-03T06:17:43.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f6c14fe60 0x7f7f6c14ba80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f6c0bd220 0x7f7f6c0a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 --2- 172.21.15.121:0/3268740224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f6c0c2860 0x7f7f6c14b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 >> 172.21.15.121:0/3268740224 conn(0x7f7f6c06c0d0 msgr2=0x7f7f6c144020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 shutdown_connections 2023-03-03T06:17:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-03T06:17:43.083+0000 7f7f71ea4700 1 -- 172.21.15.121:0/3268740224 wait complete. 2023-03-03T06:17:43.097 INFO:tasks.ceph.osd.0:Restarting daemon 2023-03-03T06:17:43.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-03-03T06:17:43.099 INFO:tasks.ceph.osd.0:Started 2023-03-03T06:17:43.100 INFO:tasks.ceph.osd.1:Restarting daemon 2023-03-03T06:17:43.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-03-03T06:17:43.101 INFO:tasks.ceph.osd.1:Started 2023-03-03T06:17:43.102 INFO:tasks.ceph.osd.2:Restarting daemon 2023-03-03T06:17:43.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-03-03T06:17:43.105 INFO:tasks.ceph.osd.2:Started 2023-03-03T06:17:43.105 INFO:tasks.ceph.osd.3:Restarting daemon 2023-03-03T06:17:43.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-03-03T06:17:43.108 INFO:tasks.ceph.osd.3:Started 2023-03-03T06:17:43.109 INFO:tasks.ceph.osd.4:Restarting daemon 2023-03-03T06:17:43.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-03-03T06:17:43.112 INFO:tasks.ceph.osd.4:Started 2023-03-03T06:17:43.112 INFO:tasks.ceph.osd.5:Restarting daemon 2023-03-03T06:17:43.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-03-03T06:17:43.116 INFO:tasks.ceph.osd.5:Started 2023-03-03T06:17:43.116 INFO:tasks.ceph.osd.6:Restarting daemon 2023-03-03T06:17:43.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-03-03T06:17:43.118 INFO:tasks.ceph.osd.6:Started 2023-03-03T06:17:43.118 INFO:tasks.ceph.osd.7:Restarting daemon 2023-03-03T06:17:43.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-03-03T06:17:43.121 INFO:tasks.ceph.osd.7:Started 2023-03-03T06:17:43.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:17:43.165 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:43.163+0000 7f1e4d65f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T06:17:43.242 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:43.241+0000 7f1e4d65f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T06:17:43.271 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.266+0000 7f4cd6c5e700 1 Processor -- start 2023-03-03T06:17:43.272 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.266+0000 7f4cd6c5e700 1 -- start start 2023-03-03T06:17:43.272 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd6c5e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd01506b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:43.273 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd6c5e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4cd006cb10 0x7f4cd00c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:43.276 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd6c5e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd0152a90 con 0x7f4cd00c4aa0 2023-03-03T06:17:43.276 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd6c5e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd00c3980 con 0x7f4cd006cb10 2023-03-03T06:17:43.276 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd6c5e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00c4620 con 0x7f4cd01531f0 2023-03-03T06:17:43.277 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd51fb700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4cd006cb10 0x7f4cd00c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:43.277 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4ccffff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4cd00c4aa0 0x7f4cd006c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:45154/0 (socket says 172.21.15.90:45154) 2023-03-03T06:17:43.277 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd51fb700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4cd006cb10 0x7f4cd00c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:47450/0 (socket says 172.21.15.90:47450) 2023-03-03T06:17:43.277 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4ccffff700 1 -- 172.21.15.90:0/1005169828 learned_addr learned my addr 172.21.15.90:0/1005169828 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:17:43.277 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd49fa700 1 --2- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd01506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:43.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd51fb700 1 -- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 msgr2=0x7f4cd01506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:43.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd51fb700 1 --2- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd01506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd51fb700 1 -- 172.21.15.90:0/1005169828 >> v1:172.21.15.121:6789/0 conn(0x7f4cd00c4aa0 legacy=0x7f4cd006c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:43.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd51fb700 1 -- 172.21.15.90:0/1005169828 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd01527b0 con 0x7f4cd006cb10 2023-03-03T06:17:43.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4cd51fb700 1 --2- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd00c3e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009d90 tx=0x7f4cc800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7bc6cbfc8211d48 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:43.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.267+0000 7f4ccf7fe700 1 -- 172.21.15.90:0/1005169828 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc800e070 con 0x7f4cd006cb10 2023-03-03T06:17:43.278 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4ccf7fe700 1 -- 172.21.15.90:0/1005169828 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc800bb20 con 0x7f4cd006cb10 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4ccf7fe700 1 -- 172.21.15.90:0/1005169828 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8012430 con 0x7f4cd006cb10 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 msgr2=0x7f4cd00c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 --2- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd00c3e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009d90 tx=0x7f4cc800b330 comp rx=0 tx=0).stop 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/1005169828 shutdown_connections 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 --2- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd00c3e10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 --2- 172.21.15.90:0/1005169828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd01506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/1005169828 >> 172.21.15.90:0/1005169828 conn(0x7f4cd00a7060 msgr2=0x7f4cd00a7460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/1005169828 shutdown_connections 2023-03-03T06:17:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.268+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/1005169828 wait complete. 2023-03-03T06:17:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.270+0000 7f4cd6c5e700 1 Processor -- start 2023-03-03T06:17:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.270+0000 7f4cd6c5e700 1 -- start start 2023-03-03T06:17:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.270+0000 7f4cd6c5e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd0148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd6c5e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4cd00c4aa0 0x7f4cd00becd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd6c5e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd00bf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd6c5e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00a95c0 con 0x7f4cd00c4aa0 2023-03-03T06:17:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd6c5e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00a8f00 con 0x7f4cd006cb10 2023-03-03T06:17:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd6c5e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01489d0 con 0x7f4cd01531f0 2023-03-03T06:17:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd0148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4ccffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4cd00c4aa0 0x7f4cd00becd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd0148490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:47452/0 (socket says 172.21.15.90:47452) 2023-03-03T06:17:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 -- 172.21.15.90:0/178180565 learned_addr learned my addr 172.21.15.90:0/178180565 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd51fb700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd00bf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 -- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 msgr2=0x7f4cd00bf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd00bf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 -- 172.21.15.90:0/178180565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4cd00c4aa0 msgr2=0x7f4cd00becd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4cd00c4aa0 0x7f4cd00becd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 -- 172.21.15.90:0/178180565 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8009a70 con 0x7f4cd006cb10 2023-03-03T06:17:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd49fa700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd0148490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400e9b0 tx=0x7f4cc400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4ccd7fa700 1 -- 172.21.15.90:0/178180565 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4009710 con 0x7f4cd006cb10 2023-03-03T06:17:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4ccd7fa700 1 -- 172.21.15.90:0/178180565 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc400ee50 con 0x7f4cd006cb10 2023-03-03T06:17:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd00c3610 con 0x7f4cd006cb10 2023-03-03T06:17:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4ccd7fa700 1 -- 172.21.15.90:0/178180565 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc400ca50 con 0x7f4cd006cb10 2023-03-03T06:17:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd00c3330 con 0x7f4cd006cb10 2023-03-03T06:17:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.271+0000 7f4ccd7fa700 1 -- 172.21.15.90:0/178180565 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f4cc4019030 con 0x7f4cd006cb10 2023-03-03T06:17:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.272+0000 7f4ccd7fa700 1 -- 172.21.15.90:0/178180565 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f4cc4006880 con 0x7f4cd006cb10 2023-03-03T06:17:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.272+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd01535c0 con 0x7f4cd006cb10 2023-03-03T06:17:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.274+0000 7f4ccd7fa700 1 -- 172.21.15.90:0/178180565 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f4cc4021060 con 0x7f4cd006cb10 2023-03-03T06:17:43.346 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.343+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cd004e750 con 0x7f4cd006cb10 2023-03-03T06:17:43.347 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.344+0000 7f4ccd7fa700 1 -- 172.21.15.90:0/178180565 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f4cc4029020 con 0x7f4cd006cb10 2023-03-03T06:17:43.347 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:17:43.347 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:17:43.077431+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:17:43.348 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 msgr2=0x7f4cd0148490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:43.348 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd0148490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400e9b0 tx=0x7f4cc400ecc0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.348 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 shutdown_connections 2023-03-03T06:17:43.349 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4cd01531f0 0x7f4cd00bf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.349 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4cd006cb10 0x7f4cd0148490 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.349 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 --2- 172.21.15.90:0/178180565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4cd00c4aa0 0x7f4cd00becd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:43.349 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 >> 172.21.15.90:0/178180565 conn(0x7f4cd00a7060 msgr2=0x7f4cd00c0350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:43.349 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 shutdown_connections 2023-03-03T06:17:43.349 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:43.345+0000 7f4cd6c5e700 1 -- 172.21.15.90:0/178180565 wait complete. 2023-03-03T06:17:43.360 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-03-03T06:17:43.360 INFO:tasks.ceph:Waiting for OSDs to come up 2023-03-03T06:17:43.706 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-03-03T06:17:43.703+0000 7f4c69fc4640 -1 Falling back to public interface 2023-03-03T06:17:43.718 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-03T06:17:43.716+0000 7f1faaa01640 -1 Falling back to public interface 2023-03-03T06:17:43.722 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-03T06:17:43.721+0000 7f2d95ea5640 -1 Falling back to public interface 2023-03-03T06:17:43.733 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-03T06:17:43.732+0000 7f40a09e6640 -1 Falling back to public interface 2023-03-03T06:17:43.739 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-03-03T06:17:43.736+0000 7fda28912640 -1 Falling back to public interface 2023-03-03T06:17:43.739 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-03-03T06:17:43.737+0000 7fa0e547d640 -1 Falling back to public interface 2023-03-03T06:17:43.747 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-03-03T06:17:43.744+0000 7fd986427640 -1 Falling back to public interface 2023-03-03T06:17:43.747 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-03T06:17:43.746+0000 7ffbf50a8640 -1 Falling back to public interface 2023-03-03T06:17:43.805 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:43.801+0000 7ff1b3fbf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T06:17:43.880 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:43.877+0000 7ff1b3fbf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T06:17:43.926 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:43.925+0000 7f1e4d65f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T06:17:44.003 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:44.001+0000 7f1e4d65f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T06:17:44.017 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:44.014+0000 7ff1b3fbf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T06:17:44.144 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:44.142+0000 7f1e4d65f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T06:17:44.166 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T06:17:44.280 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.278+0000 7f5b596f9700 1 Processor -- start 2023-03-03T06:17:44.281 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.278+0000 7f5b596f9700 1 -- start start 2023-03-03T06:17:44.283 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b596f9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5b5406be00 0x7f5b5406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:44.283 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b596f9700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b5406cf30 0x7f5b54159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:44.284 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b596f9700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b54152910 con 0x7f5b54149530 2023-03-03T06:17:44.284 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b596f9700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5406f0d0 con 0x7f5b5406cf30 2023-03-03T06:17:44.284 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b596f9700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5414fd60 con 0x7f5b5406be00 2023-03-03T06:17:44.284 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b5406cf30 0x7f5b54159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:44.285 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b5406cf30 0x7f5b54159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33320/0 (socket says 172.21.15.90:33320) 2023-03-03T06:17:44.285 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 -- 172.21.15.90:0/1344733122 learned_addr learned my addr 172.21.15.90:0/1344733122 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:44.285 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b527fc700 1 --2- 172.21.15.90:0/1344733122 >> v2:172.21.15.121:3300/0 conn(0x7f5b5406be00 0x7f5b5406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:44.286 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 -- 172.21.15.90:0/1344733122 >> v2:172.21.15.121:3300/0 conn(0x7f5b5406be00 msgr2=0x7f5b5406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:44.286 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 --2- 172.21.15.90:0/1344733122 >> v2:172.21.15.121:3300/0 conn(0x7f5b5406be00 0x7f5b5406c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.286 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 -- 172.21.15.90:0/1344733122 >> v1:172.21.15.121:6789/0 conn(0x7f5b54149530 legacy=0x7f5b5406e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:44.286 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 -- 172.21.15.90:0/1344733122 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5414f430 con 0x7f5b5406cf30 2023-03-03T06:17:44.287 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b537fe700 1 --2- 172.21.15.90:0/1344733122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b5406cf30 0x7f5b54159ca0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b540c42d0 tx=0x7f5b44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44f6a09423a4964 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:44.287 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b51ffb700 1 -- 172.21.15.90:0/1344733122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44011830 con 0x7f5b5406cf30 2023-03-03T06:17:44.287 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b51ffb700 1 -- 172.21.15.90:0/1344733122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b44011990 con 0x7f5b5406cf30 2023-03-03T06:17:44.287 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.279+0000 7f5b51ffb700 1 -- 172.21.15.90:0/1344733122 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44011d20 con 0x7f5b5406cf30 2023-03-03T06:17:44.288 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 -- 172.21.15.90:0/1344733122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b5406cf30 msgr2=0x7f5b54159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:44.288 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 --2- 172.21.15.90:0/1344733122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b5406cf30 0x7f5b54159ca0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b540c42d0 tx=0x7f5b44009bd0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.288 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 -- 172.21.15.90:0/1344733122 shutdown_connections 2023-03-03T06:17:44.288 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 --2- 172.21.15.90:0/1344733122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b5406cf30 0x7f5b54159ca0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.289 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 --2- 172.21.15.90:0/1344733122 >> v2:172.21.15.121:3300/0 conn(0x7f5b5406be00 0x7f5b5406c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.289 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 -- 172.21.15.90:0/1344733122 >> 172.21.15.90:0/1344733122 conn(0x7f5b540c3270 msgr2=0x7f5b540c3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:44.289 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 -- 172.21.15.90:0/1344733122 shutdown_connections 2023-03-03T06:17:44.289 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.280+0000 7f5b596f9700 1 -- 172.21.15.90:0/1344733122 wait complete. 2023-03-03T06:17:44.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.281+0000 7f5b596f9700 1 Processor -- start 2023-03-03T06:17:44.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.281+0000 7f5b596f9700 1 -- start start 2023-03-03T06:17:44.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b596f9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b5406be00 0x7f5b540c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:44.294 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b596f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b5406cf30 0x7f5b541504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:44.294 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b596f9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b54149530 0x7f5b541e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:44.294 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b596f9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b541d8980 con 0x7f5b54149530 2023-03-03T06:17:44.295 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b596f9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b54152110 con 0x7f5b5406cf30 2023-03-03T06:17:44.295 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b596f9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5414ebe0 con 0x7f5b5406be00 2023-03-03T06:17:44.295 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b54149530 0x7f5b541e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:44.295 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b54149530 0x7f5b541e4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33326/0 (socket says 172.21.15.90:33326) 2023-03-03T06:17:44.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 -- 172.21.15.90:0/1392271554 learned_addr learned my addr 172.21.15.90:0/1392271554 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:44.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b527fc700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b5406cf30 0x7f5b541504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:44.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b52ffd700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b5406be00 0x7f5b540c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:44.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 -- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b5406be00 msgr2=0x7f5b540c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:44.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b5406be00 0x7f5b540c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 -- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b5406cf30 msgr2=0x7f5b541504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:44.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b5406cf30 0x7f5b541504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 -- 172.21.15.90:0/1392271554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b44009550 con 0x7f5b54149530 2023-03-03T06:17:44.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.282+0000 7f5b537fe700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b54149530 0x7f5b541e4440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009c10 tx=0x7f5b44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:44.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.283+0000 7f5b3b7fe700 1 -- 172.21.15.90:0/1392271554 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44004380 con 0x7f5b54149530 2023-03-03T06:17:44.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.283+0000 7f5b3b7fe700 1 -- 172.21.15.90:0/1392271554 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4401f070 con 0x7f5b54149530 2023-03-03T06:17:44.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.283+0000 7f5b3b7fe700 1 -- 172.21.15.90:0/1392271554 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b440036a0 con 0x7f5b54149530 2023-03-03T06:17:44.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.283+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5414f9f0 con 0x7f5b54149530 2023-03-03T06:17:44.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.283+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5414f710 con 0x7f5b54149530 2023-03-03T06:17:44.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.284+0000 7f5b3b7fe700 1 -- 172.21.15.90:0/1392271554 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f5b4402a020 con 0x7f5b54149530 2023-03-03T06:17:44.300 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.284+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5406c2a0 con 0x7f5b54149530 2023-03-03T06:17:44.300 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.284+0000 7f5b3b7fe700 1 -- 172.21.15.90:0/1392271554 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5b4401ac10 con 0x7f5b54149530 2023-03-03T06:17:44.300 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.287+0000 7f5b3b7fe700 1 -- 172.21.15.90:0/1392271554 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f5b44015020 con 0x7f5b54149530 2023-03-03T06:17:44.323 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:44.320+0000 7ff1b3fbf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T06:17:44.361 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.358+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b5404e750 con 0x7f5b54149530 2023-03-03T06:17:44.362 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.358+0000 7f5b3b7fe700 1 -- 172.21.15.90:0/1392271554 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5b44003800 con 0x7f5b54149530 2023-03-03T06:17:44.362 INFO:teuthology.misc.health.smithi090.stdout: 2023-03-03T06:17:44.362 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":10,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:17:43.077431+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:17:44.363 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.359+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b54149530 msgr2=0x7f5b541e4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:44.363 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.359+0000 7f5b596f9700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b54149530 0x7f5b541e4440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009c10 tx=0x7f5b44004750 comp rx=0 tx=0).stop 2023-03-03T06:17:44.363 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.359+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 shutdown_connections 2023-03-03T06:17:44.363 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.359+0000 7f5b596f9700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b54149530 0x7f5b541e4440 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.364 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.359+0000 7f5b596f9700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b5406be00 0x7f5b540c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.364 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.359+0000 7f5b596f9700 1 --2- 172.21.15.90:0/1392271554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b5406cf30 0x7f5b541504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:44.364 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.359+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 >> 172.21.15.90:0/1392271554 conn(0x7f5b540c3270 msgr2=0x7f5b540b9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:44.364 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.360+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 shutdown_connections 2023-03-03T06:17:44.364 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:44.360+0000 7f5b596f9700 1 -- 172.21.15.90:0/1392271554 wait complete. 2023-03-03T06:17:44.373 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-03T06:17:44.443 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:44.442+0000 7f1e4d65f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T06:17:44.456 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:44.453+0000 7ff1b3fbf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T06:17:44.577 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:44.575+0000 7f1e4d65f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T06:17:44.671 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:44.668+0000 7ff1b3fbf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T06:17:44.792 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:44.790+0000 7f1e4d65f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T06:17:44.824 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:44.822+0000 7ff1b3fbf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T06:17:44.938 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:44.935+0000 7ff1b3fbf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T06:17:44.947 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:44.946+0000 7f1e4d65f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T06:17:45.062 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:45.060+0000 7f1e4d65f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T06:17:45.255 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:45.253+0000 7ff1b3fbf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T06:17:45.356 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:45.352+0000 7ff1b3fbf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T06:17:45.379 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:45.378+0000 7f1e4d65f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T06:17:45.477 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:45.476+0000 7f1e4d65f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T06:17:45.540 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:45.537+0000 7ff1b3fbf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T06:17:45.661 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:45.660+0000 7f1e4d65f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T06:17:46.053 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:46.051+0000 7ff1b3fbf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T06:17:46.186 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-03T06:17:46.185+0000 7f1e4d65f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T06:17:47.077 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:47.074+0000 7ff16b228700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-03-03T06:17:47.077 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-03-03T06:17:47.075+0000 7ff16b228700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-03-03T06:17:47.218 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-03-03T06:17:47.215+0000 7f4c69fc4640 -1 osd.1 0 log_to_monitors true 2023-03-03T06:17:47.223 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-03T06:17:47.222+0000 7f2d95ea5640 -1 osd.6 0 log_to_monitors true 2023-03-03T06:17:47.249 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-03T06:17:47.248+0000 7f1faaa01640 -1 osd.5 0 log_to_monitors true 2023-03-03T06:17:47.261 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-03T06:17:47.260+0000 7ffbf50a8640 -1 osd.7 0 log_to_monitors true 2023-03-03T06:17:47.262 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-03T06:17:47.261+0000 7f40a09e6640 -1 osd.4 0 log_to_monitors true 2023-03-03T06:17:47.291 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-03-03T06:17:47.289+0000 7fd986427640 -1 osd.3 0 log_to_monitors true 2023-03-03T06:17:47.294 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-03-03T06:17:47.292+0000 7fda28912640 -1 osd.2 0 log_to_monitors true 2023-03-03T06:17:47.296 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-03-03T06:17:47.294+0000 7fa0e547d640 -1 osd.0 0 log_to_monitors true 2023-03-03T06:17:50.953 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-03T06:17:50.950+0000 7ffbe8c5f700 -1 osd.7 0 waiting for initial osdmap 2023-03-03T06:17:50.965 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-03T06:17:50.963+0000 7ffbe103a700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.001 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-03-03T06:17:50.999+0000 7fa0d9034700 -1 osd.0 0 waiting for initial osdmap 2023-03-03T06:17:51.023 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-03-03T06:17:51.021+0000 7fa0d140f700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.087 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-03-03T06:17:51.085+0000 7f4c5db7b700 -1 osd.1 0 waiting for initial osdmap 2023-03-03T06:17:51.098 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-03-03T06:17:51.096+0000 7f4c55f56700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.105 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-03T06:17:51.102+0000 7f409459d700 -1 osd.4 0 waiting for initial osdmap 2023-03-03T06:17:51.110 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-03-03T06:17:51.108+0000 7fd979fde700 -1 osd.3 0 waiting for initial osdmap 2023-03-03T06:17:51.112 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-03T06:17:51.110+0000 7f2d89a5c700 -1 osd.6 0 waiting for initial osdmap 2023-03-03T06:17:51.116 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-03T06:17:51.112+0000 7f1f9e5b8700 -1 osd.5 0 waiting for initial osdmap 2023-03-03T06:17:51.118 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-03T06:17:51.116+0000 7f408c978700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.120 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-03-03T06:17:51.118+0000 7fd9723b9700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.131 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-03T06:17:51.129+0000 7f1f96993700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.131 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-03T06:17:51.129+0000 7f2d81e37700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.152 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-03-03T06:17:51.150+0000 7fda1c4c9700 -1 osd.2 0 waiting for initial osdmap 2023-03-03T06:17:51.166 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-03-03T06:17:51.164+0000 7fda148a4700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T06:17:51.179 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T06:17:51.288 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.285+0000 7faba1134700 1 Processor -- start 2023-03-03T06:17:51.288 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.285+0000 7faba1134700 1 -- start start 2023-03-03T06:17:51.290 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7faba1134700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0bf360 0x7fab9c06e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:51.290 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7faba1134700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fab9c0bf730 0x7fab9c151160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:51.290 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7faba1134700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c14d340 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.290 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7faba1134700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c150680 con 0x7fab9c0bf730 2023-03-03T06:17:51.290 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7faba1134700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c150960 con 0x7fab9c0bf360 2023-03-03T06:17:51.291 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9ad9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0bf360 0x7fab9c06e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:51.291 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9b59e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fab9c0bf730 0x7fab9c151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:51.291 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9a59c700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fab9c0b8cf0 0x7fab9c0b9740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:45278/0 (socket says 172.21.15.90:45278) 2023-03-03T06:17:51.291 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9b59e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fab9c0bf730 0x7fab9c151160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:47574/0 (socket says 172.21.15.90:47574) 2023-03-03T06:17:51.291 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9ad9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0bf360 0x7fab9c06e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:38364/0 (socket says 172.21.15.90:38364) 2023-03-03T06:17:51.291 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9a59c700 1 -- 172.21.15.90:0/3893436991 learned_addr learned my addr 172.21.15.90:0/3893436991 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:17:51.291 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9ad9d700 1 -- 172.21.15.90:0/3893436991 >> v2:172.21.15.121:3300/0 conn(0x7fab9c0bf730 msgr2=0x7fab9c151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:51.292 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9ad9d700 1 --2- 172.21.15.90:0/3893436991 >> v2:172.21.15.121:3300/0 conn(0x7fab9c0bf730 0x7fab9c151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.292 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9ad9d700 1 -- 172.21.15.90:0/3893436991 >> v1:172.21.15.121:6789/0 conn(0x7fab9c0b8cf0 legacy=0x7fab9c0b9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:51.292 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab99d9b700 1 -- 172.21.15.90:0/3893436991 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146812500 0 0) 0x7fab9c14d340 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.292 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9ad9d700 1 -- 172.21.15.90:0/3893436991 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c150c40 con 0x7fab9c0bf360 2023-03-03T06:17:51.292 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.286+0000 7fab9b59e700 1 --2- 172.21.15.90:0/3893436991 >> v2:172.21.15.121:3300/0 conn(0x7fab9c0bf730 0x7fab9c151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:51.292 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7fab9ad9d700 1 --2- 172.21.15.90:0/3893436991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0bf360 0x7fab9c06e9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fab90009840 tx=0x7fab90009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afaa4101667d5050 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7fab99d9b700 1 -- 172.21.15.90:0/3893436991 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90011830 con 0x7fab9c0bf360 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7fab99d9b700 1 -- 172.21.15.90:0/3893436991 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab90011990 con 0x7fab9c0bf360 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7fab99d9b700 1 -- 172.21.15.90:0/3893436991 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90011d20 con 0x7fab9c0bf360 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 -- 172.21.15.90:0/3893436991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0bf360 msgr2=0x7fab9c06e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 --2- 172.21.15.90:0/3893436991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0bf360 0x7fab9c06e9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fab90009840 tx=0x7fab90009b50 comp rx=0 tx=0).stop 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 -- 172.21.15.90:0/3893436991 shutdown_connections 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 --2- 172.21.15.90:0/3893436991 >> v2:172.21.15.121:3300/0 conn(0x7fab9c0bf730 0x7fab9c151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.293 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 --2- 172.21.15.90:0/3893436991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0bf360 0x7fab9c06e9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.294 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 -- 172.21.15.90:0/3893436991 >> 172.21.15.90:0/3893436991 conn(0x7fab9c0c3270 msgr2=0x7fab9c0c3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:51.294 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 -- 172.21.15.90:0/3893436991 shutdown_connections 2023-03-03T06:17:51.294 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.287+0000 7faba1134700 1 -- 172.21.15.90:0/3893436991 wait complete. 2023-03-03T06:17:51.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 Processor -- start 2023-03-03T06:17:51.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 -- start start 2023-03-03T06:17:51.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0b8cf0 0x7fab9c0c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:51.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab9c0bf360 0x7fab9c0a8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:51.296 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab9c0bf730 0x7fab9c0a74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:51.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0c2850 con 0x7fab9c0bf360 2023-03-03T06:17:51.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0bd3f0 con 0x7fab9c0bf730 2023-03-03T06:17:51.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7faba1134700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0bccd0 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7fab9a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab9c0bf360 0x7fab9c0a8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:51.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.289+0000 7fab9b59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab9c0bf730 0x7fab9c0a74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:51.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9ad9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0b8cf0 0x7fab9c0c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:51.297 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9b59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab9c0bf730 0x7fab9c0a74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:47578/0 (socket says 172.21.15.90:47578) 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9b59e700 1 -- 172.21.15.90:0/1495799363 learned_addr learned my addr 172.21.15.90:0/1495799363 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9ad9d700 1 -- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab9c0bf730 msgr2=0x7fab9c0a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9ad9d700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab9c0bf730 0x7fab9c0a74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9ad9d700 1 -- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab9c0bf360 msgr2=0x7fab9c0a8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9ad9d700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab9c0bf360 0x7fab9c0a8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9ad9d700 1 -- 172.21.15.90:0/1495799363 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab94009a70 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9b59e700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab9c0bf730 0x7fab9c0a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:51.298 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9a59c700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab9c0bf360 0x7fab9c0a8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:51.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab9ad9d700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0b8cf0 0x7fab9c0c2310 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab90016000 tx=0x7fab90004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:51.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab837fe700 1 -- 172.21.15.90:0/1495799363 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab900036a0 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab837fe700 1 -- 172.21.15.90:0/1495799363 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9001e070 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab900094a0 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7fab837fe700 1 -- 172.21.15.90:0/1495799363 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90003a40 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.290+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c0700a0 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.299 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.291+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c0b9240 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.300 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.291+0000 7fab837fe700 1 -- 172.21.15.90:0/1495799363 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7fab9000f050 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.300 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.292+0000 7fab837fe700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fab880682a0 0x7fab8806a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:51.300 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.292+0000 7fab837fe700 1 -- 172.21.15.90:0/1495799363 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fab90085d10 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.300 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.292+0000 7fab9a59c700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fab880682a0 0x7fab8806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:51.301 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.292+0000 7fab9a59c700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fab880682a0 0x7fab8806a750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0beb80 tx=0x7fab8c007e70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:51.301 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.297+0000 7fab837fe700 1 -- 172.21.15.90:0/1495799363 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab9008b020 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.485 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.482+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab9c14cfa0 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.486 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.483+0000 7fab837fe700 1 -- 172.21.15.90:0/1495799363 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fab90020090 con 0x7fab9c0b8cf0 2023-03-03T06:17:51.486 INFO:teuthology.misc.health.smithi090.stdout: 2023-03-03T06:17:51.486 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":12,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:17:49.072746+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:17:51.488 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fab880682a0 msgr2=0x7fab8806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:51.488 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fab880682a0 0x7fab8806a750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0beb80 tx=0x7fab8c007e70 comp rx=0 tx=0).stop 2023-03-03T06:17:51.489 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0b8cf0 msgr2=0x7fab9c0c2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:51.489 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0b8cf0 0x7fab9c0c2310 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab90016000 tx=0x7fab90004750 comp rx=0 tx=0).stop 2023-03-03T06:17:51.489 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 shutdown_connections 2023-03-03T06:17:51.489 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab9c0bf730 0x7fab9c0a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.489 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9c0b8cf0 0x7fab9c0c2310 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.489 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fab880682a0 0x7fab8806a750 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.490 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 --2- 172.21.15.90:0/1495799363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab9c0bf360 0x7fab9c0a8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:51.490 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 >> 172.21.15.90:0/1495799363 conn(0x7fab9c0c3270 msgr2=0x7fab9c0c03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:51.490 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 shutdown_connections 2023-03-03T06:17:51.490 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:51.485+0000 7faba1134700 1 -- 172.21.15.90:0/1495799363 wait complete. 2023-03-03T06:17:51.499 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-03T06:17:58.306 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T06:17:58.429 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.426+0000 7f6dfbfc2700 1 Processor -- start 2023-03-03T06:17:58.429 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.426+0000 7f6dfbfc2700 1 -- start start 2023-03-03T06:17:58.434 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6dfbfc2700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6df4065fb0 0x7f6df4150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.435 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6dfbfc2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6df406b980 0x7f6df41579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.435 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6dfbfc2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df40c0c40 con 0x7f6df40c2270 2023-03-03T06:17:58.435 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6dfbfc2700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df40c03c0 con 0x7f6df4065fb0 2023-03-03T06:17:58.435 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6dfbfc2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df406e460 con 0x7f6df406b980 2023-03-03T06:17:58.436 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df9d5e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6df40c2270 0x7f6df40bc2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52862/0 (socket says 172.21.15.90:52862) 2023-03-03T06:17:58.436 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df9d5e700 1 -- 172.21.15.90:0/1838068143 learned_addr learned my addr 172.21.15.90:0/1838068143 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:17:58.436 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6df406b980 0x7f6df41579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.437 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df955d700 1 --2- 172.21.15.90:0/1838068143 >> v2:172.21.15.121:3300/0 conn(0x7f6df4065fb0 0x7f6df4150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.437 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436947384 0 0) 0x7f6de4001110 con 0x7f6df40c2270 2023-03-03T06:17:58.437 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de00035b0 con 0x7f6df40c2270 2023-03-03T06:17:58.438 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 651551406 0 0) 0x7f6de00035b0 con 0x7f6df40c2270 2023-03-03T06:17:58.438 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6df406b980 msgr2=0x7f6df41579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:58.438 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6df406b980 0x7f6df41579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.438 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 >> v2:172.21.15.121:3300/0 conn(0x7f6df4065fb0 msgr2=0x7f6df4150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:58.439 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 --2- 172.21.15.90:0/1838068143 >> v2:172.21.15.121:3300/0 conn(0x7f6df4065fb0 0x7f6df4150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.439 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df40bf890 con 0x7f6df40c2270 2023-03-03T06:17:58.439 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f6de4001800 con 0x7f6df40c2270 2023-03-03T06:17:58.439 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.427+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 >> v1:172.21.15.90:6789/0 conn(0x7f6df40c2270 legacy=0x7f6df40bc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.440 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0003a70 0x7f6de0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.440 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0004850 0x7f6de0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.440 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de0005200 0x7f6de000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.440 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40c03c0 con 0x7f6de0005200 2023-03-03T06:17:58.441 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df406e460 con 0x7f6de0003a70 2023-03-03T06:17:58.441 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de000bf50 con 0x7f6de0004850 2023-03-03T06:17:58.441 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6df406b980 0x7f6df41579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:58.442 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df9d5e700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0003a70 0x7f6de0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.442 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de0005200 0x7f6de000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.442 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df955d700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0004850 0x7f6de0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.443 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 -- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0004850 msgr2=0x7f6de0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:58.443 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0004850 0x7f6de0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.443 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 -- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0003a70 msgr2=0x7f6de0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:58.444 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0003a70 0x7f6de0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.444 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 -- 172.21.15.90:0/1838068143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00035b0 con 0x7f6de0005200 2023-03-03T06:17:58.444 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df9d5e700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0003a70 0x7f6de0003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:17:58.444 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de0005200 0x7f6de000ba10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00e1f0 tx=0x7f6de0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43020ac56a1bbaa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:58.445 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec003c20 con 0x7f6de0005200 2023-03-03T06:17:58.445 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec00f040 con 0x7f6de0005200 2023-03-03T06:17:58.445 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.428+0000 7f6df8d5c700 1 -- 172.21.15.90:0/1838068143 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00ea30 con 0x7f6de0005200 2023-03-03T06:17:58.446 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.429+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/1838068143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de0005200 msgr2=0x7f6de000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.446 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.429+0000 7f6dfbfc2700 1 --2- 172.21.15.90:0/1838068143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de0005200 0x7f6de000ba10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00e1f0 tx=0x7f6de0005880 comp rx=0 tx=0).stop 2023-03-03T06:17:58.446 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.429+0000 7f6dfa55f700 1 -- 172.21.15.90:0/1838068143 reap_dead start 2023-03-03T06:17:58.447 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.429+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/1838068143 shutdown_connections 2023-03-03T06:17:58.447 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.429+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/1838068143 >> 172.21.15.90:0/1838068143 conn(0x7f6df40a4180 msgr2=0x7f6df40c18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:58.447 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.429+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/1838068143 shutdown_connections 2023-03-03T06:17:58.447 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.429+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/1838068143 wait complete. 2023-03-03T06:17:58.447 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.430+0000 7f6dfbfc2700 1 Processor -- start 2023-03-03T06:17:58.448 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.430+0000 7f6dfbfc2700 1 -- start start 2023-03-03T06:17:58.448 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfbfc2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0005200 0x7f6df41e8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.448 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfbfc2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df41e8590 0x7f6df41d7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.449 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfbfc2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6df41d5e30 0x7f6df41d7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.449 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfbfc2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df41e8d20 con 0x7f6df41e8590 2023-03-03T06:17:58.449 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfbfc2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df41e8a40 con 0x7f6df41d5e30 2023-03-03T06:17:58.450 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfbfc2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40a5980 con 0x7f6de0005200 2023-03-03T06:17:58.450 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df41e8590 0x7f6df41d7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.450 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df41e8590 0x7f6df41d7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33670/0 (socket says 172.21.15.90:33670) 2023-03-03T06:17:58.450 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 -- 172.21.15.90:0/3555999985 learned_addr learned my addr 172.21.15.90:0/3555999985 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:58.450 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6df41d5e30 0x7f6df41d7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.450 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df9d5e700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0005200 0x7f6df41e8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.451 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 -- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0005200 msgr2=0x7f6df41e8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.451 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0005200 0x7f6df41e8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.451 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 -- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6df41d5e30 msgr2=0x7f6df41d7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.451 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6df41d5e30 0x7f6df41d7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.452 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 -- 172.21.15.90:0/3555999985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df406b460 con 0x7f6df41e8590 2023-03-03T06:17:58.452 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfa55f700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6df41d5e30 0x7f6df41d7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:17:58.452 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df9d5e700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0005200 0x7f6df41e8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:17:58.452 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6df955d700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df41e8590 0x7f6df41d7420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6df0011aa0 tx=0x7f6df0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:58.452 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dea7fc700 1 -- 172.21.15.90:0/3555999985 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0005c30 con 0x7f6df41e8590 2023-03-03T06:17:58.453 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dea7fc700 1 -- 172.21.15.90:0/3555999985 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df0005d90 con 0x7f6df41e8590 2023-03-03T06:17:58.453 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df40bfb70 con 0x7f6df41e8590 2023-03-03T06:17:58.453 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.431+0000 7f6dea7fc700 1 -- 172.21.15.90:0/3555999985 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df000f9b0 con 0x7f6df41e8590 2023-03-03T06:17:58.453 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.432+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df406e0f0 con 0x7f6df41e8590 2023-03-03T06:17:58.455 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.432+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df41d7200 con 0x7f6df41e8590 2023-03-03T06:17:58.455 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.433+0000 7f6dea7fc700 1 -- 172.21.15.90:0/3555999985 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6df000fb10 con 0x7f6df41e8590 2023-03-03T06:17:58.455 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.433+0000 7f6dea7fc700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6dd80682f0 0x7f6dd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.456 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.433+0000 7f6dea7fc700 1 -- 172.21.15.90:0/3555999985 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6df0086140 con 0x7f6df41e8590 2023-03-03T06:17:58.456 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.433+0000 7f6df9d5e700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6dd80682f0 0x7f6dd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.456 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.433+0000 7f6df9d5e700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6dd80682f0 0x7f6dd806a7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6de4001eb0 tx=0x7f6de4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:58.456 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.438+0000 7f6dea7fc700 1 -- 172.21.15.90:0/3555999985 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6df0052e50 con 0x7f6df41e8590 2023-03-03T06:17:58.624 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.623+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df404e750 con 0x7f6df41e8590 2023-03-03T06:17:58.626 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.623+0000 7f6dea7fc700 1 -- 172.21.15.90:0/3555999985 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7f6df0056470 con 0x7f6df41e8590 2023-03-03T06:17:58.627 INFO:teuthology.misc.health.smithi090.stdout: 2023-03-03T06:17:58.627 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":16,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:17:55.095497+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:17:58.628 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6dd80682f0 msgr2=0x7f6dd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.628 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6dd80682f0 0x7f6dd806a7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6de4001eb0 tx=0x7f6de4009040 comp rx=0 tx=0).stop 2023-03-03T06:17:58.628 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df41e8590 msgr2=0x7f6df41d7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.628 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df41e8590 0x7f6df41d7420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6df0011aa0 tx=0x7f6df0011e60 comp rx=0 tx=0).stop 2023-03-03T06:17:58.628 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 shutdown_connections 2023-03-03T06:17:58.628 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6df41d5e30 0x7f6df41d7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.629 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6de0005200 0x7f6df41e8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.629 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6dd80682f0 0x7f6dd806a7a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.629 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 --2- 172.21.15.90:0/3555999985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df41e8590 0x7f6df41d7420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.629 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 >> 172.21.15.90:0/3555999985 conn(0x7f6df40a4180 msgr2=0x7f6df406d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:58.629 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 shutdown_connections 2023-03-03T06:17:58.629 INFO:teuthology.misc.health.smithi090.stderr:2023-03-03T06:17:58.625+0000 7f6dfbfc2700 1 -- 172.21.15.90:0/3555999985 wait complete. 2023-03-03T06:17:58.638 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-03-03T06:17:58.638 INFO:tasks.ceph:Creating RBD pool 2023-03-03T06:17:58.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd pool create rbd 8 2023-03-03T06:17:58.757 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd98392e700 1 Processor -- start 2023-03-03T06:17:58.757 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd98392e700 1 -- start start 2023-03-03T06:17:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd98392e700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd97c0bd670 0x7fd97c0c09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd98392e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd97c14fb00 0x7fd97c06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd98392e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0c1060 con 0x7fd97c06d9a0 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd98392e700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c06d630 con 0x7fd97c0bd670 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd98392e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c06cc90 con 0x7fd97c14fb00 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd9816ca700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd97c0bd670 0x7fd97c0c09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd9816ca700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd97c0bd670 0x7fd97c0c09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33676/0 (socket says 172.21.15.90:33676) 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.755+0000 7fd9816ca700 1 -- 172.21.15.90:0/2463590923 learned_addr learned my addr 172.21.15.90:0/2463590923 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd981ecb700 1 --2- 172.21.15.90:0/2463590923 >> v2:172.21.15.121:3300/0 conn(0x7fd97c14fb00 0x7fd97c06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd9816ca700 1 -- 172.21.15.90:0/2463590923 >> v2:172.21.15.121:3300/0 conn(0x7fd97c14fb00 msgr2=0x7fd97c06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd9816ca700 1 --2- 172.21.15.90:0/2463590923 >> v2:172.21.15.121:3300/0 conn(0x7fd97c14fb00 0x7fd97c06ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd9816ca700 1 -- 172.21.15.90:0/2463590923 >> v1:172.21.15.121:6789/0 conn(0x7fd97c06d9a0 legacy=0x7fd97c14f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd9816ca700 1 -- 172.21.15.90:0/2463590923 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c06cf70 con 0x7fd97c0bd670 2023-03-03T06:17:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd9816ca700 1 --2- 172.21.15.90:0/2463590923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c0c09a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd97c066b60 tx=0x7fd96c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9021ed83065bd54 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd973fff700 1 -- 172.21.15.90:0/2463590923 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c004230 con 0x7fd97c0bd670 2023-03-03T06:17:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd973fff700 1 -- 172.21.15.90:0/2463590923 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c004390 con 0x7fd97c0bd670 2023-03-03T06:17:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd973fff700 1 -- 172.21.15.90:0/2463590923 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0036a0 con 0x7fd97c0bd670 2023-03-03T06:17:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd98392e700 1 -- 172.21.15.90:0/2463590923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 msgr2=0x7fd97c0c09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd98392e700 1 --2- 172.21.15.90:0/2463590923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c0c09a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd97c066b60 tx=0x7fd96c009bd0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd98392e700 1 -- 172.21.15.90:0/2463590923 shutdown_connections 2023-03-03T06:17:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd98392e700 1 --2- 172.21.15.90:0/2463590923 >> v2:172.21.15.121:3300/0 conn(0x7fd97c14fb00 0x7fd97c06ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd98392e700 1 --2- 172.21.15.90:0/2463590923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c0c09a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.756+0000 7fd98392e700 1 -- 172.21.15.90:0/2463590923 >> 172.21.15.90:0/2463590923 conn(0x7fd97c0c1900 msgr2=0x7fd97c0c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:58.763 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.757+0000 7fd98392e700 1 -- 172.21.15.90:0/2463590923 shutdown_connections 2023-03-03T06:17:58.763 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.757+0000 7fd98392e700 1 -- 172.21.15.90:0/2463590923 wait complete. 2023-03-03T06:17:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 Processor -- start 2023-03-03T06:17:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 -- start start 2023-03-03T06:17:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd97c06d9a0 0x7fd97c071460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd97c14fb00 0x7fd97c1e8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1e5540 con 0x7fd97c0bd670 2023-03-03T06:17:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1e5a70 con 0x7fd97c06d9a0 2023-03-03T06:17:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd98392e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c09fc10 con 0x7fd97c14fb00 2023-03-03T06:17:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd980ec9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd980ec9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c151660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33684/0 (socket says 172.21.15.90:33684) 2023-03-03T06:17:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd980ec9700 1 -- 172.21.15.90:0/3428548615 learned_addr learned my addr 172.21.15.90:0/3428548615 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.758+0000 7fd9816ca700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd97c06d9a0 0x7fd97c071460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd981ecb700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd97c14fb00 0x7fd97c1e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd980ec9700 1 -- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd97c14fb00 msgr2=0x7fd97c1e8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd980ec9700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd97c14fb00 0x7fd97c1e8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd980ec9700 1 -- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd97c06d9a0 msgr2=0x7fd97c071460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd980ec9700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd97c06d9a0 0x7fd97c071460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd980ec9700 1 -- 172.21.15.90:0/3428548615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c009550 con 0x7fd97c0bd670 2023-03-03T06:17:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd980ec9700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c151660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd97800c500 tx=0x7fd97800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd971ffb700 1 -- 172.21.15.90:0/3428548615 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978011790 con 0x7fd97c0bd670 2023-03-03T06:17:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd971ffb700 1 -- 172.21.15.90:0/3428548615 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9780118f0 con 0x7fd97c0bd670 2023-03-03T06:17:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c0728d0 con 0x7fd97c0bd670 2023-03-03T06:17:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd971ffb700 1 -- 172.21.15.90:0/3428548615 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978011ca0 con 0x7fd97c0bd670 2023-03-03T06:17:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.759+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c06d250 con 0x7fd97c0bd670 2023-03-03T06:17:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.760+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c0bdf90 con 0x7fd97c0bd670 2023-03-03T06:17:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.760+0000 7fd971ffb700 1 -- 172.21.15.90:0/3428548615 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd978010050 con 0x7fd97c0bd670 2023-03-03T06:17:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.761+0000 7fd971ffb700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fd960068320 0x7fd96006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.761+0000 7fd971ffb700 1 -- 172.21.15.90:0/3428548615 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd9780853c0 con 0x7fd97c0bd670 2023-03-03T06:17:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.761+0000 7fd9816ca700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fd960068320 0x7fd96006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.761+0000 7fd9816ca700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fd960068320 0x7fd96006a7d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009130 tx=0x7fd96c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.766+0000 7fd971ffb700 1 -- 172.21.15.90:0/3428548615 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9780520d0 con 0x7fd97c0bd670 2023-03-03T06:17:58.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:58.949+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fd97c0719a0 con 0x7fd97c0bd670 2023-03-03T06:17:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.325+0000 7fd971ffb700 1 -- 172.21.15.90:0/3428548615 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7fd9780556f0 con 0x7fd97c0bd670 2023-03-03T06:17:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.327+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fd960068320 msgr2=0x7fd96006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.327+0000 7fd98392e700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fd960068320 0x7fd96006a7d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009130 tx=0x7fd96c008030 comp rx=0 tx=0).stop 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.327+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 msgr2=0x7fd97c151660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.327+0000 7fd98392e700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c151660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd97800c500 tx=0x7fd97800c810 comp rx=0 tx=0).stop 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 shutdown_connections 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd97c14fb00 0x7fd97c1e8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd97c06d9a0 0x7fd97c071460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fd960068320 0x7fd96006a7d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 --2- 172.21.15.90:0/3428548615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd97c0bd670 0x7fd97c151660 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 >> 172.21.15.90:0/3428548615 conn(0x7fd97c0c1900 msgr2=0x7fd97c0a68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 shutdown_connections 2023-03-03T06:17:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.328+0000 7fd98392e700 1 -- 172.21.15.90:0/3428548615 wait complete. 2023-03-03T06:17:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'rbd' created 2023-03-03T06:17:59.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-03-03T06:17:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.465+0000 7f74a43a6700 1 Processor -- start 2023-03-03T06:17:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.465+0000 7f74a43a6700 1 -- start start 2023-03-03T06:17:59.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a43a6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f749c1510d0 0x7f749c14d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:59.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a43a6700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f749c0a4d00 0x7f749c0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:59.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a43a6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c0c2da0 0x7f749c06a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:59.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a43a6700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c06e6c0 con 0x7f749c0a4d00 2023-03-03T06:17:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a43a6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c06e2b0 con 0x7f749c1510d0 2023-03-03T06:17:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a43a6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c1514a0 con 0x7f749c0c2da0 2023-03-03T06:17:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a1941700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f749c0a4d00 0x7f749c0c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a1941700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f749c0a4d00 0x7f749c0c2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33690/0 (socket says 172.21.15.90:33690) 2023-03-03T06:17:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.466+0000 7f74a1941700 1 -- 172.21.15.90:0/2926928828 learned_addr learned my addr 172.21.15.90:0/2926928828 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a2943700 1 --2- 172.21.15.90:0/2926928828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c0c2da0 0x7f749c06a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a2142700 1 --2- 172.21.15.90:0/2926928828 >> v2:172.21.15.121:3300/0 conn(0x7f749c1510d0 0x7f749c14d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1941700 1 -- 172.21.15.90:0/2926928828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c0c2da0 msgr2=0x7f749c06a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1941700 1 --2- 172.21.15.90:0/2926928828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c0c2da0 0x7f749c06a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1941700 1 -- 172.21.15.90:0/2926928828 >> v2:172.21.15.121:3300/0 conn(0x7f749c1510d0 msgr2=0x7f749c14d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1941700 1 --2- 172.21.15.90:0/2926928828 >> v2:172.21.15.121:3300/0 conn(0x7f749c1510d0 0x7f749c14d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1941700 1 -- 172.21.15.90:0/2926928828 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c072480 con 0x7f749c0a4d00 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a2943700 1 --2- 172.21.15.90:0/2926928828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c0c2da0 0x7f749c06a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1941700 1 --2- 172.21.15.90:0/2926928828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c0c2860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f749c06e1d0 tx=0x7f7498009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80381cc0c6f12035 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:59.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1140700 1 -- 172.21.15.90:0/2926928828 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749800b870 con 0x7f749c0a4d00 2023-03-03T06:17:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1140700 1 -- 172.21.15.90:0/2926928828 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749800b9d0 con 0x7f749c0a4d00 2023-03-03T06:17:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a1140700 1 -- 172.21.15.90:0/2926928828 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749800bd60 con 0x7f749c0a4d00 2023-03-03T06:17:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a43a6700 1 -- 172.21.15.90:0/2926928828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 msgr2=0x7f749c0c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.467+0000 7f74a43a6700 1 --2- 172.21.15.90:0/2926928828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c0c2860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f749c06e1d0 tx=0x7f7498009bd0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.468+0000 7f74a43a6700 1 -- 172.21.15.90:0/2926928828 shutdown_connections 2023-03-03T06:17:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.468+0000 7f74a43a6700 1 --2- 172.21.15.90:0/2926928828 >> v2:172.21.15.121:3300/0 conn(0x7f749c1510d0 0x7f749c14d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.468+0000 7f74a43a6700 1 --2- 172.21.15.90:0/2926928828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c0c2da0 0x7f749c06a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.468+0000 7f74a43a6700 1 --2- 172.21.15.90:0/2926928828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c0c2860 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.468+0000 7f74a43a6700 1 -- 172.21.15.90:0/2926928828 >> 172.21.15.90:0/2926928828 conn(0x7f749c06c0d0 msgr2=0x7f749c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:17:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.468+0000 7f74a43a6700 1 -- 172.21.15.90:0/2926928828 shutdown_connections 2023-03-03T06:17:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.468+0000 7f74a43a6700 1 -- 172.21.15.90:0/2926928828 wait complete. 2023-03-03T06:17:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 Processor -- start 2023-03-03T06:17:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 -- start start 2023-03-03T06:17:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c1e8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f749c0c2da0 0x7f749c0a70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c1510d0 0x7f749c0a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c1e8d50 con 0x7f749c0a4d00 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0a81d0 con 0x7f749c0c2da0 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a43a6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c1e88c0 con 0x7f749c1510d0 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a2142700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c1e8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.469+0000 7f74a2142700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c1e8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33694/0 (socket says 172.21.15.90:33694) 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2142700 1 -- 172.21.15.90:0/3433815164 learned_addr learned my addr 172.21.15.90:0/3433815164 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:17:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a1941700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f749c0c2da0 0x7f749c0a70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2943700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c1510d0 0x7f749c0a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2142700 1 -- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c1510d0 msgr2=0x7f749c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2142700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c1510d0 0x7f749c0a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2142700 1 -- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f749c0c2da0 msgr2=0x7f749c0a70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:17:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2142700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f749c0c2da0 0x7f749c0a70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:17:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2142700 1 -- 172.21.15.90:0/3433815164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7498009550 con 0x7f749c0a4d00 2023-03-03T06:17:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a2142700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c1e8010 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f748c00d830 tx=0x7f748c00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f7492ffd700 1 -- 172.21.15.90:0/3433815164 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c0125f0 con 0x7f749c0a4d00 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f7492ffd700 1 -- 172.21.15.90:0/3433815164 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c012750 con 0x7f749c0a4d00 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c14d0d0 con 0x7f749c0a4d00 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f7492ffd700 1 -- 172.21.15.90:0/3433815164 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c012b00 con 0x7f749c0a4d00 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.470+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c06e910 con 0x7f749c0a4d00 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.471+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c14d3b0 con 0x7f749c0a4d00 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.472+0000 7f7492ffd700 1 -- 172.21.15.90:0/3433815164 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f748c012c60 con 0x7f749c0a4d00 2023-03-03T06:17:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.472+0000 7f7492ffd700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74800682f0 0x7f748006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:17:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.472+0000 7f74a1941700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74800682f0 0x7f748006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:17:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.472+0000 7f7492ffd700 1 -- 172.21.15.90:0/3433815164 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f748c085a40 con 0x7f749c0a4d00 2023-03-03T06:17:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.473+0000 7f74a1941700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74800682f0 0x7f748006a7a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f749c0a5140 tx=0x7f7498008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:17:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.477+0000 7f7492ffd700 1 -- 172.21.15.90:0/3433815164 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f748c052600 con 0x7f749c0a4d00 2023-03-03T06:17:59.664 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:17:59.662+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f749c04e7c0 con 0x7f749c0a4d00 2023-03-03T06:18:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.339+0000 7f7492ffd700 1 -- 172.21.15.90:0/3433815164 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f748c055c20 con 0x7f749c0a4d00 2023-03-03T06:18:00.343 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.341+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74800682f0 msgr2=0x7f748006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.343 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.341+0000 7f74a43a6700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74800682f0 0x7f748006a7a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f749c0a5140 tx=0x7f7498008030 comp rx=0 tx=0).stop 2023-03-03T06:18:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.341+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 msgr2=0x7f749c1e8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.341+0000 7f74a43a6700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c1e8010 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f748c00d830 tx=0x7f748c00db40 comp rx=0 tx=0).stop 2023-03-03T06:18:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 shutdown_connections 2023-03-03T06:18:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f749c1510d0 0x7f749c0a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74800682f0 0x7f748006a7a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f749c0a4d00 0x7f749c1e8010 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 --2- 172.21.15.90:0/3433815164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f749c0c2da0 0x7f749c0a70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 >> 172.21.15.90:0/3433815164 conn(0x7f749c06c0d0 msgr2=0x7f749c0bcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 shutdown_connections 2023-03-03T06:18:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.342+0000 7f74a43a6700 1 -- 172.21.15.90:0/3433815164 wait complete. 2023-03-03T06:18:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:enabled application 'rbd' on pool 'rbd' 2023-03-03T06:18:00.358 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-03-03T06:18:00.359 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T06:18:00.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T06:18:00.361 INFO:tasks.ceph.mds.a:Started 2023-03-03T06:18:00.361 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T06:18:00.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T06:18:00.364 INFO:tasks.ceph.mds.c:Started 2023-03-03T06:18:00.364 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T06:18:00.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T06:18:00.367 INFO:tasks.ceph.mds.b:Started 2023-03-03T06:18:00.368 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T06:18:00.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T06:18:00.370 INFO:tasks.ceph.mds.d:Started 2023-03-03T06:18:00.371 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-03-03T06:18:00.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:00.444 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2023-03-03T06:18:00.444 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2023-03-03T06:18:00.445 INFO:tasks.ceph.mds.a.smithi090.stderr:2023-03-03T06:18:00.443+0000 7f80f6c17700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-03-03T06:18:00.453 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2023-03-03T06:18:00.454 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2023-03-03T06:18:00.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.502+0000 7f04f0595700 1 Processor -- start 2023-03-03T06:18:00.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.502+0000 7f04f0595700 1 -- start start 2023-03-03T06:18:00.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04f0595700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e806c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04f0595700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04e806a330 0x7f04e80c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04f0595700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e814fb40 con 0x7f04e80c37e0 2023-03-03T06:18:00.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04f0595700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e80c0060 con 0x7f04e806a330 2023-03-03T06:18:00.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04f0595700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e80b8d00 con 0x7f04e806bcf0 2023-03-03T06:18:00.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04e806a330 0x7f04e80c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04e806a330 0x7f04e80c32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33728/0 (socket says 172.21.15.90:33728) 2023-03-03T06:18:00.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 -- 172.21.15.90:0/654588339 learned_addr learned my addr 172.21.15.90:0/654588339 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:00.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04eeb32700 1 --1- 172.21.15.90:0/654588339 >> v1:172.21.15.90:6789/0 conn(0x7f04e80c37e0 0x7f04e81526b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52938/0 (socket says 172.21.15.90:52938) 2023-03-03T06:18:00.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04ee331700 1 --2- 172.21.15.90:0/654588339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e806c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 -- 172.21.15.90:0/654588339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 msgr2=0x7f04e806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:00.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 --2- 172.21.15.90:0/654588339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e806c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 -- 172.21.15.90:0/654588339 >> v1:172.21.15.90:6789/0 conn(0x7f04e80c37e0 legacy=0x7f04e81526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04ed32f700 1 -- 172.21.15.90:0/654588339 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346306646 0 0) 0x7f04e814fb40 con 0x7f04e80c37e0 2023-03-03T06:18:00.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 -- 172.21.15.90:0/654588339 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e80c3c20 con 0x7f04e806a330 2023-03-03T06:18:00.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04ee331700 1 --2- 172.21.15.90:0/654588339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e806c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:00.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04edb30700 1 --2- 172.21.15.90:0/654588339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e806a330 0x7f04e80c32a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04d80098f0 tx=0x7f04d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671f6ed564cefc3d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:00.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04ed32f700 1 -- 172.21.15.90:0/654588339 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d800b8a0 con 0x7f04e806a330 2023-03-03T06:18:00.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04ed32f700 1 -- 172.21.15.90:0/654588339 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d800ba00 con 0x7f04e806a330 2023-03-03T06:18:00.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.503+0000 7f04ed32f700 1 -- 172.21.15.90:0/654588339 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d800bd90 con 0x7f04e806a330 2023-03-03T06:18:00.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 -- 172.21.15.90:0/654588339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e806a330 msgr2=0x7f04e80c32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 --2- 172.21.15.90:0/654588339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e806a330 0x7f04e80c32a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04d80098f0 tx=0x7f04d8009c00 comp rx=0 tx=0).stop 2023-03-03T06:18:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 -- 172.21.15.90:0/654588339 shutdown_connections 2023-03-03T06:18:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 --2- 172.21.15.90:0/654588339 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e806c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 --2- 172.21.15.90:0/654588339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e806a330 0x7f04e80c32a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 -- 172.21.15.90:0/654588339 >> 172.21.15.90:0/654588339 conn(0x7f04e80a6f40 msgr2=0x7f04e80a7340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 -- 172.21.15.90:0/654588339 shutdown_connections 2023-03-03T06:18:00.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.504+0000 7f04f0595700 1 -- 172.21.15.90:0/654588339 wait complete. 2023-03-03T06:18:00.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.505+0000 7f04f0595700 1 Processor -- start 2023-03-03T06:18:00.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.505+0000 7f04f0595700 1 -- start start 2023-03-03T06:18:00.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04f0595700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04e806a330 0x7f04e80a95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04f0595700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e80b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04f0595700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e80c37e0 0x7f04e80b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04f0595700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e80aac50 con 0x7f04e80c37e0 2023-03-03T06:18:00.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04f0595700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e8147e70 con 0x7f04e806a330 2023-03-03T06:18:00.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04f0595700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e80a67b0 con 0x7f04e806bcf0 2023-03-03T06:18:00.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04eeb32700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e80c37e0 0x7f04e80b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04eeb32700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e80c37e0 0x7f04e80b8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33738/0 (socket says 172.21.15.90:33738) 2023-03-03T06:18:00.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.506+0000 7f04eeb32700 1 -- 172.21.15.90:0/2661061691 learned_addr learned my addr 172.21.15.90:0/2661061691 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:00.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04ee331700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04e806a330 0x7f04e80a95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04edb30700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e80b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04eeb32700 1 -- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 msgr2=0x7f04e80b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04eeb32700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e80b7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04eeb32700 1 -- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04e806a330 msgr2=0x7f04e80a95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04eeb32700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04e806a330 0x7f04e80a95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04eeb32700 1 -- 172.21.15.90:0/2661061691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d8009550 con 0x7f04e80c37e0 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04eeb32700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e80c37e0 0x7f04e80b8130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f04e000cba0 tx=0x7f04e000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04deffd700 1 -- 172.21.15.90:0/2661061691 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e000ab80 con 0x7f04e80c37e0 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04deffd700 1 -- 172.21.15.90:0/2661061691 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e000ad00 con 0x7f04e80c37e0 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e814fd90 con 0x7f04e80c37e0 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04deffd700 1 -- 172.21.15.90:0/2661061691 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0007310 con 0x7f04e80c37e0 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.507+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e80c0300 con 0x7f04e80c37e0 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.508+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e80c51b0 con 0x7f04e80c37e0 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.508+0000 7f04deffd700 1 -- 172.21.15.90:0/2661061691 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f04e001a030 con 0x7f04e80c37e0 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.509+0000 7f04deffd700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f04cc0682f0 0x7f04cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.509+0000 7f04deffd700 1 -- 172.21.15.90:0/2661061691 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f04e0085690 con 0x7f04e80c37e0 2023-03-03T06:18:00.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.509+0000 7f04ee331700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f04cc0682f0 0x7f04cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.509+0000 7f04ee331700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f04cc0682f0 0x7f04cc06a7a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f04e4006fd0 tx=0x7f04e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:00.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.514+0000 7f04deffd700 1 -- 172.21.15.90:0/2661061691 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04e00521e0 con 0x7f04e80c37e0 2023-03-03T06:18:00.700 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.698+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04e80aadc0 con 0x7f04e80c37e0 2023-03-03T06:18:00.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.699+0000 7f04deffd700 1 -- 172.21.15.90:0/2661061691 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14181 (secure 0 0 0) 0x7f04e80aadc0 con 0x7f04e80c37e0 2023-03-03T06:18:00.701 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:00.701 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:00.330068+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:00.703 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f04cc0682f0 msgr2=0x7f04cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.704 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f04cc0682f0 0x7f04cc06a7a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f04e4006fd0 tx=0x7f04e4008040 comp rx=0 tx=0).stop 2023-03-03T06:18:00.704 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e80c37e0 msgr2=0x7f04e80b8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.704 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e80c37e0 0x7f04e80b8130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f04e000cba0 tx=0x7f04e000cf60 comp rx=0 tx=0).stop 2023-03-03T06:18:00.704 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 shutdown_connections 2023-03-03T06:18:00.704 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f04cc0682f0 0x7f04cc06a7a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.704 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04e80c37e0 0x7f04e80b8130 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.705 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04e806a330 0x7f04e80a95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.705 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.701+0000 7f04f0595700 1 --2- 172.21.15.90:0/2661061691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04e806bcf0 0x7f04e80b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.705 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.702+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 >> 172.21.15.90:0/2661061691 conn(0x7f04e80a6f40 msgr2=0x7f04e81d84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:00.705 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.702+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 shutdown_connections 2023-03-03T06:18:00.705 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.702+0000 7f04f0595700 1 -- 172.21.15.90:0/2661061691 wait complete. 2023-03-03T06:18:00.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T06:17:52.113684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T06:17:58.969014+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T06:18:00.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T06:18:00.838 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa4ae82700 1 Processor -- start 2023-03-03T06:18:00.839 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa4ae82700 1 -- start start 2023-03-03T06:18:00.840 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa4ae82700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffa44147030 0x7ffa4414d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.841 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa4ae82700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa44072500 con 0x7ffa440bddc0 2023-03-03T06:18:00.841 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa4ae82700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa440ba280 con 0x7ffa440b9da0 2023-03-03T06:18:00.841 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa4ae82700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa440b8820 con 0x7ffa44147030 2023-03-03T06:18:00.841 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa43fff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffa440bddc0 0x7ffa440b9690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52948/0 (socket says 172.21.15.90:52948) 2023-03-03T06:18:00.841 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.837+0000 7ffa43fff700 1 -- 172.21.15.90:0/2339435226 learned_addr learned my addr 172.21.15.90:0/2339435226 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:00.841 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa48c1e700 1 --2- 172.21.15.90:0/2339435226 >> v2:172.21.15.121:3300/0 conn(0x7ffa44147030 0x7ffa4414d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.842 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa437fe700 1 -- 172.21.15.90:0/2339435226 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842366059 0 0) 0x7ffa44072500 con 0x7ffa440bddc0 2023-03-03T06:18:00.842 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa437fe700 1 -- 172.21.15.90:0/2339435226 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa340035b0 con 0x7ffa440bddc0 2023-03-03T06:18:00.842 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa437fe700 1 -- 172.21.15.90:0/2339435226 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3943411451 0 0) 0x7ffa340035b0 con 0x7ffa440bddc0 2023-03-03T06:18:00.842 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa48c1e700 1 -- 172.21.15.90:0/2339435226 >> v1:172.21.15.121:6789/0 conn(0x7ffa440b9da0 legacy=0x7ffa44157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.842 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa48c1e700 1 -- 172.21.15.90:0/2339435226 >> v1:172.21.15.90:6789/0 conn(0x7ffa440bddc0 legacy=0x7ffa440b9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.842 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa48c1e700 1 -- 172.21.15.90:0/2339435226 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa44072220 con 0x7ffa44147030 2023-03-03T06:18:00.842 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa48c1e700 1 --2- 172.21.15.90:0/2339435226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa44147030 0x7ffa4414d7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009840 tx=0x7ffa38009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9eb97000c621605 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa437fe700 1 -- 172.21.15.90:0/2339435226 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30001670 con 0x7ffa44147030 2023-03-03T06:18:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa437fe700 1 -- 172.21.15.90:0/2339435226 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa38004030 con 0x7ffa44147030 2023-03-03T06:18:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.838+0000 7ffa437fe700 1 -- 172.21.15.90:0/2339435226 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa380043a0 con 0x7ffa44147030 2023-03-03T06:18:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.839+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2339435226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa44147030 msgr2=0x7ffa4414d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.839+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2339435226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa44147030 0x7ffa4414d7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009840 tx=0x7ffa38009b50 comp rx=0 tx=0).stop 2023-03-03T06:18:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.839+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2339435226 shutdown_connections 2023-03-03T06:18:00.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.839+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2339435226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa44147030 0x7ffa4414d7a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.839+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2339435226 >> 172.21.15.90:0/2339435226 conn(0x7ffa440c3230 msgr2=0x7ffa440c3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:00.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.839+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2339435226 shutdown_connections 2023-03-03T06:18:00.844 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.839+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2339435226 wait complete. 2023-03-03T06:18:00.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.840+0000 7ffa4ae82700 1 Processor -- start 2023-03-03T06:18:00.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.840+0000 7ffa4ae82700 1 -- start start 2023-03-03T06:18:00.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa4ae82700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa440b9da0 0x7ffa4414bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa4ae82700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa440bddc0 0x7ffa440c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa4ae82700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa44147030 0x7ffa441e5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa4ae82700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa441451d0 con 0x7ffa440b9da0 2023-03-03T06:18:00.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa4ae82700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa44144140 con 0x7ffa440bddc0 2023-03-03T06:18:00.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa4ae82700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4414c410 con 0x7ffa44147030 2023-03-03T06:18:00.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa48c1e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa440b9da0 0x7ffa4414bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa48c1e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa440b9da0 0x7ffa4414bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33748/0 (socket says 172.21.15.90:33748) 2023-03-03T06:18:00.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.841+0000 7ffa48c1e700 1 -- 172.21.15.90:0/2544617224 learned_addr learned my addr 172.21.15.90:0/2544617224 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:00.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa4941f700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa44147030 0x7ffa441e5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.848 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa43fff700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa440bddc0 0x7ffa440c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.848 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa48c1e700 1 -- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa44147030 msgr2=0x7ffa441e5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.848 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa48c1e700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa44147030 0x7ffa441e5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.848 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa48c1e700 1 -- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa440bddc0 msgr2=0x7ffa440c10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:00.848 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa48c1e700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa440bddc0 0x7ffa440c10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:00.848 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa48c1e700 1 -- 172.21.15.90:0/2544617224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa380094a0 con 0x7ffa440b9da0 2023-03-03T06:18:00.848 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa4941f700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa44147030 0x7ffa441e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa48c1e700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa440b9da0 0x7ffa4414bed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa38000c00 tx=0x7ffa38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa417fa700 1 -- 172.21.15.90:0/2544617224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38003df0 con 0x7ffa440b9da0 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa417fa700 1 -- 172.21.15.90:0/2544617224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3801f070 con 0x7ffa440b9da0 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa417fa700 1 -- 172.21.15.90:0/2544617224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38011870 con 0x7ffa440b9da0 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4414d280 con 0x7ffa440b9da0 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.842+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4414cfa0 con 0x7ffa440b9da0 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.843+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa44070020 con 0x7ffa440b9da0 2023-03-03T06:18:00.849 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.844+0000 7ffa417fa700 1 -- 172.21.15.90:0/2544617224 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffa3802a020 con 0x7ffa440b9da0 2023-03-03T06:18:00.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.844+0000 7ffa417fa700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ffa280682f0 0x7ffa2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:00.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.844+0000 7ffa417fa700 1 -- 172.21.15.90:0/2544617224 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffa38086f60 con 0x7ffa440b9da0 2023-03-03T06:18:00.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.844+0000 7ffa43fff700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ffa280682f0 0x7ffa2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:00.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.844+0000 7ffa43fff700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ffa280682f0 0x7ffa2806a7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffa30001eb0 tx=0x7ffa30001c60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:00.850 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:00.848+0000 7ffa417fa700 1 -- 172.21.15.90:0/2544617224 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa380b85e0 con 0x7ffa440b9da0 2023-03-03T06:18:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.030+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa440706d0 con 0x7ffa440b9da0 2023-03-03T06:18:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.030+0000 7ffa417fa700 1 -- 172.21.15.90:0/2544617224 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa38053b10 con 0x7ffa440b9da0 2023-03-03T06:18:01.033 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2023-03-03T06:18:01.034 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ffa280682f0 msgr2=0x7ffa2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.035 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ffa280682f0 0x7ffa2806a7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffa30001eb0 tx=0x7ffa30001c60 comp rx=0 tx=0).stop 2023-03-03T06:18:01.035 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa440b9da0 msgr2=0x7ffa4414bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.035 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa440b9da0 0x7ffa4414bed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa38000c00 tx=0x7ffa38004750 comp rx=0 tx=0).stop 2023-03-03T06:18:01.035 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 shutdown_connections 2023-03-03T06:18:01.035 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa44147030 0x7ffa441e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.036 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ffa280682f0 0x7ffa2806a7a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.036 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa440b9da0 0x7ffa4414bed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.036 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 --2- 172.21.15.90:0/2544617224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa440bddc0 0x7ffa440c10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.036 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 >> 172.21.15.90:0/2544617224 conn(0x7ffa440c3230 msgr2=0x7ffa4414ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:01.036 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 shutdown_connections 2023-03-03T06:18:01.036 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.033+0000 7ffa4ae82700 1 -- 172.21.15.90:0/2544617224 wait complete. 2023-03-03T06:18:01.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T06:18:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.166+0000 7f406187e700 1 Processor -- start 2023-03-03T06:18:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f406187e700 1 -- start start 2023-03-03T06:18:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f406187e700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f405c06a330 0x7f405c0c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f406187e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f405c0c3e20 0x7f405c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f406187e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0c3450 con 0x7f405c06bcd0 2023-03-03T06:18:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f406187e700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405c06c7b0 con 0x7f405c06a330 2023-03-03T06:18:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f406187e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405c0c4260 con 0x7f405c0c3e20 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f405a7fc700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f405c06a330 0x7f405c0c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f405a7fc700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f405c06a330 0x7f405c0c38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33758/0 (socket says 172.21.15.90:33758) 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f405a7fc700 1 -- 172.21.15.90:0/238847129 learned_addr learned my addr 172.21.15.90:0/238847129 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.167+0000 7f405b7fe700 1 --2- 172.21.15.90:0/238847129 >> v2:172.21.15.121:3300/0 conn(0x7f405c0c3e20 0x7f405c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f405a7fc700 1 -- 172.21.15.90:0/238847129 >> v2:172.21.15.121:3300/0 conn(0x7f405c0c3e20 msgr2=0x7f405c159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f405a7fc700 1 --2- 172.21.15.90:0/238847129 >> v2:172.21.15.121:3300/0 conn(0x7f405c0c3e20 0x7f405c159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f405a7fc700 1 -- 172.21.15.90:0/238847129 >> v1:172.21.15.90:6789/0 conn(0x7f405c06bcd0 legacy=0x7f405c06c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f405a7fc700 1 -- 172.21.15.90:0/238847129 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c0c4e30 con 0x7f405c06a330 2023-03-03T06:18:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f4059ffb700 1 -- 172.21.15.90:0/238847129 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957383191 0 0) 0x7f405c0c3450 con 0x7f405c06bcd0 2023-03-03T06:18:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f405a7fc700 1 --2- 172.21.15.90:0/238847129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06a330 0x7f405c0c38e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f405c0bfa20 tx=0x7f404c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6406550fa75d33d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f4059ffb700 1 -- 172.21.15.90:0/238847129 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00b870 con 0x7f405c06a330 2023-03-03T06:18:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f4059ffb700 1 -- 172.21.15.90:0/238847129 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c00b9d0 con 0x7f405c06a330 2023-03-03T06:18:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f4059ffb700 1 -- 172.21.15.90:0/238847129 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00bd60 con 0x7f405c06a330 2023-03-03T06:18:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 -- 172.21.15.90:0/238847129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06a330 msgr2=0x7f405c0c38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 --2- 172.21.15.90:0/238847129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06a330 0x7f405c0c38e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f405c0bfa20 tx=0x7f404c009bd0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.174 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 -- 172.21.15.90:0/238847129 shutdown_connections 2023-03-03T06:18:01.174 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 --2- 172.21.15.90:0/238847129 >> v2:172.21.15.121:3300/0 conn(0x7f405c0c3e20 0x7f405c159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.174 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 --2- 172.21.15.90:0/238847129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06a330 0x7f405c0c38e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.174 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 -- 172.21.15.90:0/238847129 >> 172.21.15.90:0/238847129 conn(0x7f405c0a6f50 msgr2=0x7f405c0a7350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:01.174 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 -- 172.21.15.90:0/238847129 shutdown_connections 2023-03-03T06:18:01.174 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.168+0000 7f406187e700 1 -- 172.21.15.90:0/238847129 wait complete. 2023-03-03T06:18:01.174 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.169+0000 7f406187e700 1 Processor -- start 2023-03-03T06:18:01.176 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f406187e700 1 -- start start 2023-03-03T06:18:01.176 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f406187e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f405c06a330 0x7f405c0a8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f406187e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06bcd0 0x7f405c14a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f406187e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405c0c3e20 0x7f405c0aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f406187e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c0a9ec0 con 0x7f405c06bcd0 2023-03-03T06:18:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f406187e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c0bb030 con 0x7f405c06a330 2023-03-03T06:18:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f406187e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405c0a9080 con 0x7f405c0c3e20 2023-03-03T06:18:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f405a7fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06bcd0 0x7f405c14a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f405a7fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06bcd0 0x7f405c14a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33764/0 (socket says 172.21.15.90:33764) 2023-03-03T06:18:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f405a7fc700 1 -- 172.21.15.90:0/611880707 learned_addr learned my addr 172.21.15.90:0/611880707 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.170+0000 7f405affd700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f405c06a330 0x7f405c0a8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405b7fe700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405c0c3e20 0x7f405c0aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405a7fc700 1 -- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405c0c3e20 msgr2=0x7f405c0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405a7fc700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405c0c3e20 0x7f405c0aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405a7fc700 1 -- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f405c06a330 msgr2=0x7f405c0a8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405a7fc700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f405c06a330 0x7f405c0a8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405a7fc700 1 -- 172.21.15.90:0/611880707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c009550 con 0x7f405c06bcd0 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405b7fe700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405c0c3e20 0x7f405c0aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f405a7fc700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06bcd0 0x7f405c14a7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f404c014040 tx=0x7f404c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f4043fff700 1 -- 172.21.15.90:0/611880707 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c004310 con 0x7f405c06bcd0 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f4043fff700 1 -- 172.21.15.90:0/611880707 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c01e070 con 0x7f405c06bcd0 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c14fb00 con 0x7f405c06bcd0 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f4043fff700 1 -- 172.21.15.90:0/611880707 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c0036a0 con 0x7f405c06bcd0 2023-03-03T06:18:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.171+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c0c5110 con 0x7f405c06bcd0 2023-03-03T06:18:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.172+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c14fde0 con 0x7f405c06bcd0 2023-03-03T06:18:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.172+0000 7f4043fff700 1 -- 172.21.15.90:0/611880707 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f404c029020 con 0x7f405c06bcd0 2023-03-03T06:18:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.173+0000 7f4043fff700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f40480682f0 0x7f404806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.173+0000 7f4043fff700 1 -- 172.21.15.90:0/611880707 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f404c086aa0 con 0x7f405c06bcd0 2023-03-03T06:18:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.173+0000 7f405affd700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f40480682f0 0x7f404806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.173+0000 7f405affd700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f40480682f0 0x7f404806a7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4050000f60 tx=0x7f4050009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.177+0000 7f4043fff700 1 -- 172.21.15.90:0/611880707 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f404c053650 con 0x7f405c06bcd0 2023-03-03T06:18:01.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.359+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f405c0aa000 con 0x7f405c06bcd0 2023-03-03T06:18:01.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.360+0000 7f4043fff700 1 -- 172.21.15.90:0/611880707 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f404c015020 con 0x7f405c06bcd0 2023-03-03T06:18:01.362 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 8 2023-03-03T06:18:01.363 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f40480682f0 msgr2=0x7f404806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.363 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f40480682f0 0x7f404806a7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4050000f60 tx=0x7f4050009040 comp rx=0 tx=0).stop 2023-03-03T06:18:01.364 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06bcd0 msgr2=0x7f405c14a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.364 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06bcd0 0x7f405c14a7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f404c014040 tx=0x7f404c004750 comp rx=0 tx=0).stop 2023-03-03T06:18:01.364 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 shutdown_connections 2023-03-03T06:18:01.364 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405c0c3e20 0x7f405c0aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.364 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f405c06a330 0x7f405c0a8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.364 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f40480682f0 0x7f404806a7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.365 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 --2- 172.21.15.90:0/611880707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f405c06bcd0 0x7f405c14a7c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.365 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 >> 172.21.15.90:0/611880707 conn(0x7f405c0a6f50 msgr2=0x7f405c1db5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:01.365 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 shutdown_connections 2023-03-03T06:18:01.365 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.362+0000 7f406187e700 1 -- 172.21.15.90:0/611880707 wait complete. 2023-03-03T06:18:01.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:18:01.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.494+0000 7f37f8892700 1 Processor -- start 2023-03-03T06:18:01.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.494+0000 7f37f8892700 1 -- start start 2023-03-03T06:18:01.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f8892700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 0x7f37f415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f8892700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f4153340 con 0x7f37f406ea90 2023-03-03T06:18:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f8892700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f41497f0 con 0x7f37f4149420 2023-03-03T06:18:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f8892700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f4152750 con 0x7f37f414f7d0 2023-03-03T06:18:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f1d9b700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f37f406ea90 0x7f37f414f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52980/0 (socket says 172.21.15.90:52980) 2023-03-03T06:18:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f1d9b700 1 -- 172.21.15.90:0/2405360427 learned_addr learned my addr 172.21.15.90:0/2405360427 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:01.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 0x7f37f415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849542524 0 0) 0x7f37f4153340 con 0x7f37f406ea90 2023-03-03T06:18:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37dc0035b0 con 0x7f37f406ea90 2023-03-03T06:18:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836770826 0 0) 0x7f37f41497f0 con 0x7f37f4149420 2023-03-03T06:18:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f4153340 con 0x7f37f4149420 2023-03-03T06:18:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258772057 0 0) 0x7f37dc0035b0 con 0x7f37f406ea90 2023-03-03T06:18:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 msgr2=0x7f37f415aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 0x7f37f415aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 >> v1:172.21.15.121:6789/0 conn(0x7f37f4149420 legacy=0x7f37f40c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4152a10 con 0x7f37f406ea90 2023-03-03T06:18:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f37e8001670 con 0x7f37f406ea90 2023-03-03T06:18:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 0x7f37f415aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 >> v1:172.21.15.90:6789/0 conn(0x7f37f406ea90 legacy=0x7f37f414f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37dc003c30 0x7f37dc004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 0x7f37dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37dc004d90 0x7f37dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f4152750 con 0x7f37dc004d90 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc00bbd0 con 0x7f37dc003c30 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc00bd90 con 0x7f37f414f7d0 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37dc004d90 0x7f37dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f259c700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37dc003c30 0x7f37dc004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.495+0000 7f37f1d9b700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 0x7f37dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f2d9d700 1 -- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 msgr2=0x7f37dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37f414f7d0 0x7f37dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f2d9d700 1 -- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37dc003c30 msgr2=0x7f37dc004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37dc003c30 0x7f37dc004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f2d9d700 1 -- 172.21.15.90:0/2405360427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec009a70 con 0x7f37dc004d90 2023-03-03T06:18:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37dc004d90 0x7f37dc005200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00e350 tx=0x7f37dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e93d6d01daa41cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec00edf0 con 0x7f37dc004d90 2023-03-03T06:18:01.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec00f040 con 0x7f37dc004d90 2023-03-03T06:18:01.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f159a700 1 -- 172.21.15.90:0/2405360427 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0146a0 con 0x7f37dc004d90 2023-03-03T06:18:01.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f8892700 1 -- 172.21.15.90:0/2405360427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37dc004d90 msgr2=0x7f37dc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f8892700 1 --2- 172.21.15.90:0/2405360427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37dc004d90 0x7f37dc005200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00e350 tx=0x7f37dc005880 comp rx=0 tx=0).stop 2023-03-03T06:18:01.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.496+0000 7f37f2d9d700 1 -- 172.21.15.90:0/2405360427 reap_dead start 2023-03-03T06:18:01.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.497+0000 7f37f8892700 1 -- 172.21.15.90:0/2405360427 shutdown_connections 2023-03-03T06:18:01.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.497+0000 7f37f8892700 1 -- 172.21.15.90:0/2405360427 >> 172.21.15.90:0/2405360427 conn(0x7f37f406c0d0 msgr2=0x7f37f40c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:01.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.497+0000 7f37f8892700 1 -- 172.21.15.90:0/2405360427 shutdown_connections 2023-03-03T06:18:01.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.497+0000 7f37f8892700 1 -- 172.21.15.90:0/2405360427 wait complete. 2023-03-03T06:18:01.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 Processor -- start 2023-03-03T06:18:01.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 -- start start 2023-03-03T06:18:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37dc004d90 0x7f37f4147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37f413f100 0x7f37f41d8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f414cf30 0x7f37f41e82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41403e0 con 0x7f37f413f100 2023-03-03T06:18:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f414eda0 con 0x7f37f414cf30 2023-03-03T06:18:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.498+0000 7f37f8892700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41d9110 con 0x7f37dc004d90 2023-03-03T06:18:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37f413f100 0x7f37f41d8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37f413f100 0x7f37f41d8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33788/0 (socket says 172.21.15.90:33788) 2023-03-03T06:18:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 -- 172.21.15.90:0/3171279963 learned_addr learned my addr 172.21.15.90:0/3171279963 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f259c700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37dc004d90 0x7f37f4147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f414cf30 0x7f37f41e82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 -- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37dc004d90 msgr2=0x7f37f4147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37dc004d90 0x7f37f4147260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 -- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f414cf30 msgr2=0x7f37f41e82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f414cf30 0x7f37f41e82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 -- 172.21.15.90:0/3171279963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4152fd0 con 0x7f37f413f100 2023-03-03T06:18:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f2d9d700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f414cf30 0x7f37f41e82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f1d9b700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37f413f100 0x7f37f41d8bd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37e8004fd0 tx=0x7f37e8007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37e2ffd700 1 -- 172.21.15.90:0/3171279963 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8011580 con 0x7f37f413f100 2023-03-03T06:18:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37e2ffd700 1 -- 172.21.15.90:0/3171279963 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e80116e0 con 0x7f37f413f100 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f4152cf0 con 0x7f37f413f100 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37e2ffd700 1 -- 172.21.15.90:0/3171279963 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8011a50 con 0x7f37f413f100 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.499+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f406e610 con 0x7f37f413f100 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.500+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f414ef10 con 0x7f37f413f100 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.500+0000 7f37e2ffd700 1 -- 172.21.15.90:0/3171279963 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37e8011bb0 con 0x7f37f413f100 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.501+0000 7f37e2ffd700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f37d4068320 0x7f37d406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.501+0000 7f37e2ffd700 1 -- 172.21.15.90:0/3171279963 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f37e8085a90 con 0x7f37f413f100 2023-03-03T06:18:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.501+0000 7f37f259c700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f37d4068320 0x7f37d406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.501+0000 7f37f259c700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f37d4068320 0x7f37d406a7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37e4007fd0 tx=0x7f37e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.505+0000 7f37e2ffd700 1 -- 172.21.15.90:0/3171279963 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37e80525e0 con 0x7f37f413f100 2023-03-03T06:18:01.731 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.730+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37f404e750 con 0x7f37f413f100 2023-03-03T06:18:01.732 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.730+0000 7f37e2ffd700 1 -- 172.21.15.90:0/3171279963 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3530 (secure 0 0 0) 0x7f37e8055c00 con 0x7f37f413f100 2023-03-03T06:18:01.732 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:01.733 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1770068404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1770068404},{"type":"v1","addr":"172.21.15.121:6834","nonce":1770068404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1411382210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":1411382210},{"type":"v1","addr":"172.21.15.90:6837","nonce":1411382210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4290,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2684520267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2684520267},{"type":"v1","addr":"172.21.15.121:6835","nonce":2684520267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/3337533052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3337533052},{"type":"v1","addr":"172.21.15.90:6836","nonce":3337533052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-03-03T06:18:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.732+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f37d4068320 msgr2=0x7f37d406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.732+0000 7f37f8892700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f37d4068320 0x7f37d406a7d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37e4007fd0 tx=0x7f37e4009040 comp rx=0 tx=0).stop 2023-03-03T06:18:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.732+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37f413f100 msgr2=0x7f37f41d8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.735 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.732+0000 7f37f8892700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37f413f100 0x7f37f41d8bd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f37e8004fd0 tx=0x7f37e8007340 comp rx=0 tx=0).stop 2023-03-03T06:18:01.735 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 shutdown_connections 2023-03-03T06:18:01.735 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f414cf30 0x7f37f41e82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.735 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37dc004d90 0x7f37f4147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.735 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f37d4068320 0x7f37d406a7d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.735 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 --2- 172.21.15.90:0/3171279963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37f413f100 0x7f37f41d8bd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.735 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 >> 172.21.15.90:0/3171279963 conn(0x7f37f406c0d0 msgr2=0x7f37f4151db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 shutdown_connections 2023-03-03T06:18:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.733+0000 7f37f8892700 1 -- 172.21.15.90:0/3171279963 wait complete. 2023-03-03T06:18:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 3 2023-03-03T06:18:01.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:01.864 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.863+0000 7f89334b3700 1 Processor -- start 2023-03-03T06:18:01.865 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.863+0000 7f89334b3700 1 -- start start 2023-03-03T06:18:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f89334b3700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f892c06dbe0 0x7f892c0bcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f89334b3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c0bde80 0x7f892c14fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f89334b3700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f892c072160 0x7f892c072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f89334b3700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c14f230 con 0x7f892c072160 2023-03-03T06:18:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f89334b3700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c150160 con 0x7f892c06dbe0 2023-03-03T06:18:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f89334b3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c0bd590 con 0x7f892c0bde80 2023-03-03T06:18:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f892c072160 0x7f892c072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f892c072160 0x7f892c072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33796/0 (socket says 172.21.15.90:33796) 2023-03-03T06:18:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 -- 172.21.15.90:0/137157761 learned_addr learned my addr 172.21.15.90:0/137157761 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f893124f700 1 --2- 172.21.15.90:0/137157761 >> v2:172.21.15.121:3300/0 conn(0x7f892c06dbe0 0x7f892c0bcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8930a4e700 1 --2- 172.21.15.90:0/137157761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c0bde80 0x7f892c14fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 -- 172.21.15.90:0/137157761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c0bde80 msgr2=0x7f892c14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 --2- 172.21.15.90:0/137157761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c0bde80 0x7f892c14fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 -- 172.21.15.90:0/137157761 >> v2:172.21.15.121:3300/0 conn(0x7f892c06dbe0 msgr2=0x7f892c0bcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 --2- 172.21.15.90:0/137157761 >> v2:172.21.15.121:3300/0 conn(0x7f892c06dbe0 0x7f892c0bcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 -- 172.21.15.90:0/137157761 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0bd870 con 0x7f892c072160 2023-03-03T06:18:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8930a4e700 1 --2- 172.21.15.90:0/137157761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c0bde80 0x7f892c14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:01.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8931a50700 1 --2- 172.21.15.90:0/137157761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c072160 0x7f892c072590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f892c0c1860 tx=0x7f892400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309749d2f6830522 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8923fff700 1 -- 172.21.15.90:0/137157761 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892400e070 con 0x7f892c072160 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8923fff700 1 -- 172.21.15.90:0/137157761 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892400b9c0 con 0x7f892c072160 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.864+0000 7f8923fff700 1 -- 172.21.15.90:0/137157761 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892400bd70 con 0x7f892c072160 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 -- 172.21.15.90:0/137157761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c072160 msgr2=0x7f892c072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 --2- 172.21.15.90:0/137157761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c072160 0x7f892c072590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f892c0c1860 tx=0x7f892400b330 comp rx=0 tx=0).stop 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 -- 172.21.15.90:0/137157761 shutdown_connections 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 --2- 172.21.15.90:0/137157761 >> v2:172.21.15.121:3300/0 conn(0x7f892c06dbe0 0x7f892c0bcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 --2- 172.21.15.90:0/137157761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c072160 0x7f892c072590 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 --2- 172.21.15.90:0/137157761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c0bde80 0x7f892c14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 -- 172.21.15.90:0/137157761 >> 172.21.15.90:0/137157761 conn(0x7f892c0c1900 msgr2=0x7f892c0c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:01.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 -- 172.21.15.90:0/137157761 shutdown_connections 2023-03-03T06:18:01.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.865+0000 7f89334b3700 1 -- 172.21.15.90:0/137157761 wait complete. 2023-03-03T06:18:01.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.866+0000 7f89334b3700 1 Processor -- start 2023-03-03T06:18:01.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.866+0000 7f89334b3700 1 -- start start 2023-03-03T06:18:01.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f89334b3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c06dbe0 0x7f892c146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f89334b3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c072160 0x7f892c145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f89334b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f892c0bde80 0x7f892c0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f89334b3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c145a70 con 0x7f892c06dbe0 2023-03-03T06:18:01.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f89334b3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c148280 con 0x7f892c0bde80 2023-03-03T06:18:01.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f89334b3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c147500 con 0x7f892c072160 2023-03-03T06:18:01.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c06dbe0 0x7f892c146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c06dbe0 0x7f892c146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33798/0 (socket says 172.21.15.90:33798) 2023-03-03T06:18:01.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 -- 172.21.15.90:0/1365587310 learned_addr learned my addr 172.21.15.90:0/1365587310 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:01.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f8930a4e700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c072160 0x7f892c145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f8931a50700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f892c0bde80 0x7f892c0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 -- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c072160 msgr2=0x7f892c145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c072160 0x7f892c145530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 -- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f892c0bde80 msgr2=0x7f892c0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:01.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f892c0bde80 0x7f892c0a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:01.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 -- 172.21.15.90:0/1365587310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8924009b30 con 0x7f892c06dbe0 2023-03-03T06:18:01.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f8931a50700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f892c0bde80 0x7f892c0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:01.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f893124f700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c06dbe0 0x7f892c146f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f892800e810 tx=0x7f892800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.867+0000 7f8921ffb700 1 -- 172.21.15.90:0/1365587310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928013610 con 0x7f892c06dbe0 2023-03-03T06:18:01.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.868+0000 7f8921ffb700 1 -- 172.21.15.90:0/1365587310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8928013770 con 0x7f892c06dbe0 2023-03-03T06:18:01.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.868+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c14f700 con 0x7f892c06dbe0 2023-03-03T06:18:01.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.868+0000 7f8921ffb700 1 -- 172.21.15.90:0/1365587310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928013b20 con 0x7f892c06dbe0 2023-03-03T06:18:01.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.868+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c14f420 con 0x7f892c06dbe0 2023-03-03T06:18:01.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.869+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c06d610 con 0x7f892c06dbe0 2023-03-03T06:18:01.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.869+0000 7f8921ffb700 1 -- 172.21.15.90:0/1365587310 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8928023070 con 0x7f892c06dbe0 2023-03-03T06:18:01.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.870+0000 7f8921ffb700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f89100682f0 0x7f891006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:01.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.870+0000 7f8921ffb700 1 -- 172.21.15.90:0/1365587310 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8928085ab0 con 0x7f892c06dbe0 2023-03-03T06:18:01.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.870+0000 7f8930a4e700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f89100682f0 0x7f891006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:01.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.870+0000 7f8930a4e700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f89100682f0 0x7f891006a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f891c006fd0 tx=0x7f891c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:01.879 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:01.873+0000 7f8921ffb700 1 -- 172.21.15.90:0/1365587310 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8928052660 con 0x7f892c06dbe0 2023-03-03T06:18:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.058+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f892c04e750 con 0x7f892c06dbe0 2023-03-03T06:18:02.061 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.059+0000 7f8921ffb700 1 -- 172.21.15.90:0/1365587310 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14168 (secure 0 0 0) 0x7f8928021090 con 0x7f892c06dbe0 2023-03-03T06:18:02.061 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:02.061 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":19,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:01.336933+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f89100682f0 msgr2=0x7f891006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f89100682f0 0x7f891006a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f891c006fd0 tx=0x7f891c008040 comp rx=0 tx=0).stop 2023-03-03T06:18:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c06dbe0 msgr2=0x7f892c146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c06dbe0 0x7f892c146f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f892800e810 tx=0x7f892800ebd0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 shutdown_connections 2023-03-03T06:18:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f892c0bde80 0x7f892c0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f89100682f0 0x7f891006a7a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c06dbe0 0x7f892c146f20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 --2- 172.21.15.90:0/1365587310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f892c072160 0x7f892c145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.061+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 >> 172.21.15.90:0/1365587310 conn(0x7f892c0c1900 msgr2=0x7f892c14da10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.062+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 shutdown_connections 2023-03-03T06:18:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.062+0000 7f89334b3700 1 -- 172.21.15.90:0/1365587310 wait complete. 2023-03-03T06:18:02.075 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T06:17:52.113684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T06:17:58.969014+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T06:18:02.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T06:18:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.195+0000 7fc7d7a7c700 1 Processor -- start 2023-03-03T06:18:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.195+0000 7fc7d7a7c700 1 -- start start 2023-03-03T06:18:02.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d7a7c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc7d00c3920 0x7fc7d00c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d7a7c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00b8e70 0x7fc7d0157b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d7a7c700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d01507c0 con 0x7fc7d01474b0 2023-03-03T06:18:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d7a7c700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d006ff40 con 0x7fc7d00c3920 2023-03-03T06:18:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d7a7c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00b92f0 con 0x7fc7d00b8e70 2023-03-03T06:18:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d5017700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc7d01474b0 0x7fc7d014d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53012/0 (socket says 172.21.15.90:53012) 2023-03-03T06:18:02.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d5017700 1 -- 172.21.15.90:0/188273366 learned_addr learned my addr 172.21.15.90:0/188273366 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:02.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d6019700 1 --2- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00b8e70 0x7fc7d0157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d5818700 1 --2- 172.21.15.90:0/188273366 >> v2:172.21.15.121:3300/0 conn(0x7fc7d00c3920 0x7fc7d00c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d4816700 1 -- 172.21.15.90:0/188273366 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179859264 0 0) 0x7fc7d01507c0 con 0x7fc7d01474b0 2023-03-03T06:18:02.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.196+0000 7fc7d4816700 1 -- 172.21.15.90:0/188273366 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7bc0035b0 con 0x7fc7d01474b0 2023-03-03T06:18:02.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d5818700 1 -- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00b8e70 msgr2=0x7fc7d0157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:02.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d5818700 1 --2- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00b8e70 0x7fc7d0157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d4816700 1 -- 172.21.15.90:0/188273366 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 192125059 0 0) 0x7fc7bc0035b0 con 0x7fc7d01474b0 2023-03-03T06:18:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d5818700 1 -- 172.21.15.90:0/188273366 >> v1:172.21.15.90:6789/0 conn(0x7fc7d01474b0 legacy=0x7fc7d014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d5818700 1 -- 172.21.15.90:0/188273366 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d006fc60 con 0x7fc7d00c3920 2023-03-03T06:18:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d5818700 1 --2- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00c3920 0x7fc7d00c3cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009840 tx=0x7fc7c0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28cfc0361b6a622 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d4816700 1 -- 172.21.15.90:0/188273366 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc001670 con 0x7fc7d00c3920 2023-03-03T06:18:02.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d4816700 1 -- 172.21.15.90:0/188273366 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c0004030 con 0x7fc7d00c3920 2023-03-03T06:18:02.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d4816700 1 -- 172.21.15.90:0/188273366 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00043a0 con 0x7fc7d00c3920 2023-03-03T06:18:02.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00c3920 msgr2=0x7fc7d00c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.197+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00c3920 0x7fc7d00c3cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009840 tx=0x7fc7c0009b50 comp rx=0 tx=0).stop 2023-03-03T06:18:02.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.198+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/188273366 shutdown_connections 2023-03-03T06:18:02.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.198+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00b8e70 0x7fc7d0157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.198+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/188273366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00c3920 0x7fc7d00c3cf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.198+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/188273366 >> 172.21.15.90:0/188273366 conn(0x7fc7d006afc0 msgr2=0x7fc7d00c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.198+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/188273366 shutdown_connections 2023-03-03T06:18:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.198+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/188273366 wait complete. 2023-03-03T06:18:02.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 Processor -- start 2023-03-03T06:18:02.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 -- start start 2023-03-03T06:18:02.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00b8e70 0x7fc7d01e5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00c3920 0x7fc7d01d7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d01d7bb0 0x7fc7d00bb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d0147bd0 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00baae0 con 0x7fc7d00b8e70 2023-03-03T06:18:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d7a7c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00bad60 con 0x7fc7d00c3920 2023-03-03T06:18:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d6019700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d01d7bb0 0x7fc7d00bb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d6019700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d01d7bb0 0x7fc7d00bb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33814/0 (socket says 172.21.15.90:33814) 2023-03-03T06:18:02.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.199+0000 7fc7d6019700 1 -- 172.21.15.90:0/3385992270 learned_addr learned my addr 172.21.15.90:0/3385992270 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d5017700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00c3920 0x7fc7d01d7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d6019700 1 -- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00c3920 msgr2=0x7fc7d01d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d5818700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00b8e70 0x7fc7d01e5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d6019700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00c3920 0x7fc7d01d7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d6019700 1 -- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00b8e70 msgr2=0x7fc7d01e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d6019700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00b8e70 0x7fc7d01e5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d6019700 1 -- 172.21.15.90:0/3385992270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00094a0 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d6019700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d01d7bb0 0x7fc7d00bb630 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d30 tx=0x7fc7c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7c67fc700 1 -- 172.21.15.90:0/3385992270 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8009ba0 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7c67fc700 1 -- 172.21.15.90:0/3385992270 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c800f040 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7c67fc700 1 -- 172.21.15.90:0/3385992270 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8014480 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d014d3e0 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.200+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d00bf740 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.201+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d006f740 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.202+0000 7fc7c67fc700 1 -- 172.21.15.90:0/3385992270 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7c80145e0 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.203+0000 7fc7c67fc700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fc7b4068320 0x7fc7b406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.203+0000 7fc7c67fc700 1 -- 172.21.15.90:0/3385992270 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc7c8086c30 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.203+0000 7fc7d5818700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fc7b4068320 0x7fc7b406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.203+0000 7fc7d5818700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fc7b4068320 0x7fc7b406a7d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00090b0 tx=0x7fc7c0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.206+0000 7fc7c67fc700 1 -- 172.21.15.90:0/3385992270 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7c80536f0 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.392 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.390+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7d00bd470 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.392 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.391+0000 7fc7c67fc700 1 -- 172.21.15.90:0/3385992270 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7c8018020 con 0x7fc7d01d7bb0 2023-03-03T06:18:02.393 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2023-03-03T06:18:02.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fc7b4068320 msgr2=0x7fc7b406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fc7b4068320 0x7fc7b406a7d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00090b0 tx=0x7fc7c0008030 comp rx=0 tx=0).stop 2023-03-03T06:18:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d01d7bb0 msgr2=0x7fc7d00bb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d01d7bb0 0x7fc7d00bb630 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d30 tx=0x7fc7c800c3a0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 shutdown_connections 2023-03-03T06:18:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fc7b4068320 0x7fc7b406a7d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d01d7bb0 0x7fc7d00bb630 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7d00b8e70 0x7fc7d01e5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 --2- 172.21.15.90:0/3385992270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7d00c3920 0x7fc7d01d7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 >> 172.21.15.90:0/3385992270 conn(0x7fc7d006afc0 msgr2=0x7fc7d014d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 shutdown_connections 2023-03-03T06:18:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.393+0000 7fc7d7a7c700 1 -- 172.21.15.90:0/3385992270 wait complete. 2023-03-03T06:18:02.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T06:18:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.528+0000 7f829c2ed700 1 Processor -- start 2023-03-03T06:18:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.528+0000 7f829c2ed700 1 -- start start 2023-03-03T06:18:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.528+0000 7f829c2ed700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8294149420 0x7f82940c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829c2ed700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8294153340 con 0x7f829406ea90 2023-03-03T06:18:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829c2ed700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82941497f0 con 0x7f82940c2da0 2023-03-03T06:18:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829c2ed700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8294152750 con 0x7f8294149420 2023-03-03T06:18:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a089700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8294149420 0x7f82940c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a089700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8294149420 0x7f82940c2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33818/0 (socket says 172.21.15.90:33818) 2023-03-03T06:18:02.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a88a700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f829406ea90 0x7f8294159a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53028/0 (socket says 172.21.15.90:53028) 2023-03-03T06:18:02.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a089700 1 -- 172.21.15.90:0/901601399 learned_addr learned my addr 172.21.15.90:0/901601399 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:02.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a089700 1 -- 172.21.15.90:0/901601399 >> v1:172.21.15.121:6789/0 conn(0x7f82940c2da0 legacy=0x7f829414f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a089700 1 -- 172.21.15.90:0/901601399 >> v1:172.21.15.90:6789/0 conn(0x7f829406ea90 legacy=0x7f8294159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a089700 1 -- 172.21.15.90:0/901601399 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8294152a10 con 0x7f8294149420 2023-03-03T06:18:02.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f829a089700 1 --2- 172.21.15.90:0/901601399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8294149420 0x7f82940c2860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82900098f0 tx=0x7f8290009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1226d4771e2421e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f8299087700 1 -- 172.21.15.90:0/901601399 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000b8a0 con 0x7f8294149420 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f8299087700 1 -- 172.21.15.90:0/901601399 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829000ba00 con 0x7f8294149420 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.529+0000 7f8299087700 1 -- 172.21.15.90:0/901601399 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000bd90 con 0x7f8294149420 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.530+0000 7f829c2ed700 1 -- 172.21.15.90:0/901601399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8294149420 msgr2=0x7f82940c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.530+0000 7f829c2ed700 1 --2- 172.21.15.90:0/901601399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8294149420 0x7f82940c2860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82900098f0 tx=0x7f8290009c00 comp rx=0 tx=0).stop 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.530+0000 7f829c2ed700 1 -- 172.21.15.90:0/901601399 shutdown_connections 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.530+0000 7f829c2ed700 1 --2- 172.21.15.90:0/901601399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8294149420 0x7f82940c2860 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.530+0000 7f829c2ed700 1 -- 172.21.15.90:0/901601399 >> 172.21.15.90:0/901601399 conn(0x7f829406c0d0 msgr2=0x7f82940c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.530+0000 7f829c2ed700 1 -- 172.21.15.90:0/901601399 shutdown_connections 2023-03-03T06:18:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.530+0000 7f829c2ed700 1 -- 172.21.15.90:0/901601399 wait complete. 2023-03-03T06:18:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.531+0000 7f829c2ed700 1 Processor -- start 2023-03-03T06:18:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.531+0000 7f829c2ed700 1 -- start start 2023-03-03T06:18:02.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829406ea90 0x7f8294147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82940c2da0 0x7f82941e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8294149420 0x7f82941e5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829414c260 con 0x7f82940c2da0 2023-03-03T06:18:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829414b810 con 0x7f8294149420 2023-03-03T06:18:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8294148370 con 0x7f829406ea90 2023-03-03T06:18:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82940c2da0 0x7f82941e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82940c2da0 0x7f82941e4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33826/0 (socket says 172.21.15.90:33826) 2023-03-03T06:18:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 -- 172.21.15.90:0/2205110271 learned_addr learned my addr 172.21.15.90:0/2205110271 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829a88a700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8294149420 0x7f82941e5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829a089700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829406ea90 0x7f8294147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 -- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829406ea90 msgr2=0x7f8294147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829406ea90 0x7f8294147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 -- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8294149420 msgr2=0x7f82941e5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8294149420 0x7f82941e5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 -- 172.21.15.90:0/2205110271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8290009550 con 0x7f82940c2da0 2023-03-03T06:18:02.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829a089700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829406ea90 0x7f8294147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:02.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f8299888700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82940c2da0 0x7f82941e4f00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f828400c4e0 tx=0x7f828400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f828affd700 1 -- 172.21.15.90:0/2205110271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840052d0 con 0x7f82940c2da0 2023-03-03T06:18:02.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f828affd700 1 -- 172.21.15.90:0/2205110271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284005430 con 0x7f82940c2da0 2023-03-03T06:18:02.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8294152fd0 con 0x7f82940c2da0 2023-03-03T06:18:02.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f828affd700 1 -- 172.21.15.90:0/2205110271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840038c0 con 0x7f82940c2da0 2023-03-03T06:18:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.532+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8294152cf0 con 0x7f82940c2da0 2023-03-03T06:18:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.533+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829406e610 con 0x7f82940c2da0 2023-03-03T06:18:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.534+0000 7f828affd700 1 -- 172.21.15.90:0/2205110271 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8284019030 con 0x7f82940c2da0 2023-03-03T06:18:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.534+0000 7f828affd700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f82780682f0 0x7f827806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.534+0000 7f829a089700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f82780682f0 0x7f827806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.534+0000 7f828affd700 1 -- 172.21.15.90:0/2205110271 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8284084be0 con 0x7f82940c2da0 2023-03-03T06:18:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.534+0000 7f829a089700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f82780682f0 0x7f827806a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8290014040 tx=0x7f8290008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.538+0000 7f828affd700 1 -- 172.21.15.90:0/2205110271 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8284051790 con 0x7f82940c2da0 2023-03-03T06:18:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.720+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f829404e750 con 0x7f82940c2da0 2023-03-03T06:18:02.722 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.720+0000 7f828affd700 1 -- 172.21.15.90:0/2205110271 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f82840170a0 con 0x7f82940c2da0 2023-03-03T06:18:02.722 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 8 2023-03-03T06:18:02.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f82780682f0 msgr2=0x7f827806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f82780682f0 0x7f827806a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8290014040 tx=0x7f8290008030 comp rx=0 tx=0).stop 2023-03-03T06:18:02.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82940c2da0 msgr2=0x7f82941e4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.725 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82940c2da0 0x7f82941e4f00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f828400c4e0 tx=0x7f828400c7f0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 shutdown_connections 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8294149420 0x7f82941e5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829406ea90 0x7f8294147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f82780682f0 0x7f827806a7a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 --2- 172.21.15.90:0/2205110271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82940c2da0 0x7f82941e4f00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 >> 172.21.15.90:0/2205110271 conn(0x7f829406c0d0 msgr2=0x7f8294150280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 shutdown_connections 2023-03-03T06:18:02.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.722+0000 7f829c2ed700 1 -- 172.21.15.90:0/2205110271 wait complete. 2023-03-03T06:18:02.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T06:18:02.854 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.852+0000 7f207d1fc700 1 Processor -- start 2023-03-03T06:18:02.855 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207d1fc700 1 -- start start 2023-03-03T06:18:02.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207d1fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2078149420 0x7f20780c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.857 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207d1fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 0x7f207814f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207d1fc700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2078153340 con 0x7f207806ea90 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207d1fc700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20781497f0 con 0x7f2078149420 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207d1fc700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2078152750 con 0x7f20780c2da0 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207759e700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f207806ea90 0x7f207806a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53042/0 (socket says 172.21.15.90:53042) 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.853+0000 7f207759e700 1 -- 172.21.15.90:0/2857282363 learned_addr learned my addr 172.21.15.90:0/2857282363 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94988582 0 0) 0x7f2078153340 con 0x7f207806ea90 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2076d9d700 1 --2- 172.21.15.90:0/2857282363 >> v2:172.21.15.121:3300/0 conn(0x7f2078149420 0x7f20780c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.858 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 0x7f207814f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20600035b0 con 0x7f207806ea90 2023-03-03T06:18:02.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539324539 0 0) 0x7f20600035b0 con 0x7f207806ea90 2023-03-03T06:18:02.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 msgr2=0x7f207814f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:02.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 0x7f207814f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 >> v2:172.21.15.121:3300/0 conn(0x7f2078149420 msgr2=0x7f20780c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:02.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 --2- 172.21.15.90:0/2857282363 >> v2:172.21.15.121:3300/0 conn(0x7f2078149420 0x7f20780c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.859 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078152a10 con 0x7f207806ea90 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f20700019f0 con 0x7f207806ea90 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 >> v1:172.21.15.90:6789/0 conn(0x7f207806ea90 legacy=0x7f207806a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060003a70 0x7f2060003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2060004850 0x7f2060004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 0x7f2060005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20781497f0 con 0x7f2060004850 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2078152750 con 0x7f2060003a70 2023-03-03T06:18:02.860 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206000ba40 con 0x7f20780c2da0 2023-03-03T06:18:02.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2060004850 0x7f2060004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f2076d9d700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060003a70 0x7f2060003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207759e700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 0x7f2060005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 -- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 msgr2=0x7f2060005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:02.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c2da0 0x7f2060005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 -- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060003a70 msgr2=0x7f2060003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:02.861 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060003a70 0x7f2060003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 -- 172.21.15.90:0/2857282363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20600035b0 con 0x7f2060004850 2023-03-03T06:18:02.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.854+0000 7f207659c700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2060004850 0x7f2060004cc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2068009b30 tx=0x7f2060005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b371fd7ab3abfbd0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206800e6c0 con 0x7f2060004850 2023-03-03T06:18:02.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206800e820 con 0x7f2060004850 2023-03-03T06:18:02.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f2075d9b700 1 -- 172.21.15.90:0/2857282363 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206800eb90 con 0x7f2060004850 2023-03-03T06:18:02.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f207d1fc700 1 -- 172.21.15.90:0/2857282363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2060004850 msgr2=0x7f2060004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.862 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f207d1fc700 1 --2- 172.21.15.90:0/2857282363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2060004850 0x7f2060004cc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2068009b30 tx=0x7f2060005880 comp rx=0 tx=0).stop 2023-03-03T06:18:02.863 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f207759e700 1 -- 172.21.15.90:0/2857282363 reap_dead start 2023-03-03T06:18:02.863 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f207d1fc700 1 -- 172.21.15.90:0/2857282363 shutdown_connections 2023-03-03T06:18:02.863 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f207d1fc700 1 -- 172.21.15.90:0/2857282363 >> 172.21.15.90:0/2857282363 conn(0x7f207806c0d0 msgr2=0x7f20780c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:02.863 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f207d1fc700 1 -- 172.21.15.90:0/2857282363 shutdown_connections 2023-03-03T06:18:02.863 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.855+0000 7f207d1fc700 1 -- 172.21.15.90:0/2857282363 wait complete. 2023-03-03T06:18:02.866 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.857+0000 7f207d1fc700 1 Processor -- start 2023-03-03T06:18:02.866 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.857+0000 7f207d1fc700 1 -- start start 2023-03-03T06:18:02.866 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207d1fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060004850 0x7f20780a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.866 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207d1fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20780a78c0 0x7f20780a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.866 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207d1fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c0de0 0x7f20780c1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.866 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207d1fc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2078144340 con 0x7f20780a78c0 2023-03-03T06:18:02.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207d1fc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20780c1790 con 0x7f2060004850 2023-03-03T06:18:02.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207d1fc700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20780c0900 con 0x7f20780c0de0 2023-03-03T06:18:02.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20780a78c0 0x7f20780a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20780a78c0 0x7f20780a7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33846/0 (socket says 172.21.15.90:33846) 2023-03-03T06:18:02.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 -- 172.21.15.90:0/349440810 learned_addr learned my addr 172.21.15.90:0/349440810 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:02.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207759e700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c0de0 0x7f20780c1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f2076d9d700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060004850 0x7f20780a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 -- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c0de0 msgr2=0x7f20780c1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c0de0 0x7f20780c1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 -- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060004850 msgr2=0x7f20780a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:02.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060004850 0x7f20780a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:02.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 -- 172.21.15.90:0/349440810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078152fd0 con 0x7f20780a78c0 2023-03-03T06:18:02.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f2076d9d700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060004850 0x7f20780a6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:02.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207659c700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20780a78c0 0x7f20780a7d30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2060004780 tx=0x7f20680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f205f7fe700 1 -- 172.21.15.90:0/349440810 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2068021070 con 0x7f20780a78c0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f205f7fe700 1 -- 172.21.15.90:0/349440810 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2068004550 con 0x7f20780a78c0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2078152cf0 con 0x7f20780a78c0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.858+0000 7f205f7fe700 1 -- 172.21.15.90:0/349440810 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206800e6c0 con 0x7f20780a78c0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.859+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207806e610 con 0x7f20780a78c0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.860+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207804e750 con 0x7f20780a78c0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.860+0000 7f205f7fe700 1 -- 172.21.15.90:0/349440810 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f206802e030 con 0x7f20780a78c0 2023-03-03T06:18:02.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.860+0000 7f205f7fe700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f20640682f0 0x7f206406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.860+0000 7f205f7fe700 1 -- 172.21.15.90:0/349440810 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2068086c00 con 0x7f20780a78c0 2023-03-03T06:18:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.860+0000 7f2076d9d700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f20640682f0 0x7f206406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.860+0000 7f2076d9d700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f20640682f0 0x7f206406a7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f206c009bd0 tx=0x7f206c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:02.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:02.864+0000 7f205f7fe700 1 -- 172.21.15.90:0/349440810 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f206808c020 con 0x7f20780a78c0 2023-03-03T06:18:03.052 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.051+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f20780c0ac0 con 0x7f20780a78c0 2023-03-03T06:18:03.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.051+0000 7f205f7fe700 1 -- 172.21.15.90:0/349440810 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7f2068017090 con 0x7f20780a78c0 2023-03-03T06:18:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:03.054 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2023-03-03T06:18:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-03-03T06:18:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2023-03-03T06:18:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2023-03-03T06:18:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-03-03T06:18:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-03-03T06:18:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 2, 2023-03-03T06:18:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": "rbd" 2023-03-03T06:18:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-03-03T06:18:03.055 INFO:teuthology.orchestra.run.smithi090.stdout:] 2023-03-03T06:18:03.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f20640682f0 msgr2=0x7f206406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:03.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f20640682f0 0x7f206406a7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f206c009bd0 tx=0x7f206c008040 comp rx=0 tx=0).stop 2023-03-03T06:18:03.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20780a78c0 msgr2=0x7f20780a7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:03.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20780a78c0 0x7f20780a7d30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2060004780 tx=0x7f20680047c0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 shutdown_connections 2023-03-03T06:18:03.056 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20780c0de0 0x7f20780c1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2060004850 0x7f20780a6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f20640682f0 0x7f206406a7a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 --2- 172.21.15.90:0/349440810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20780a78c0 0x7f20780a7d30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 >> 172.21.15.90:0/349440810 conn(0x7f207806c0d0 msgr2=0x7f20780bee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:03.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 shutdown_connections 2023-03-03T06:18:03.057 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.053+0000 7f207d1fc700 1 -- 172.21.15.90:0/349440810 wait complete. 2023-03-03T06:18:03.066 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-03T06:18:03.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-03T06:18:03.190 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.188+0000 7f23c065e700 1 Processor -- start 2023-03-03T06:18:03.190 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.188+0000 7f23c065e700 1 -- start start 2023-03-03T06:18:03.196 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23c065e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 0x7f23b806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23c065e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f23b80c4ae0 0x7f23b806c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23c065e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80c50a0 con 0x7f23b8149620 2023-03-03T06:18:03.197 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23c065e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b806b890 con 0x7f23b80c4ae0 2023-03-03T06:18:03.198 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23c065e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b814fbb0 con 0x7f23b80c3f60 2023-03-03T06:18:03.198 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23be3fa700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f23b8149620 0x7f23b80c3850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53058/0 (socket says 172.21.15.90:53058) 2023-03-03T06:18:03.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23be3fa700 1 -- 172.21.15.90:0/1475941001 learned_addr learned my addr 172.21.15.90:0/1475941001 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:03.199 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874800930 0 0) 0x7f23b80c50a0 con 0x7f23b8149620 2023-03-03T06:18:03.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bdbf9700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 0x7f23b806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a40035b0 con 0x7f23b8149620 2023-03-03T06:18:03.200 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bebfb700 1 --2- 172.21.15.90:0/1475941001 >> v2:172.21.15.121:3300/0 conn(0x7f23b80c4ae0 0x7f23b806c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909345791 0 0) 0x7f23a40035b0 con 0x7f23b8149620 2023-03-03T06:18:03.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 msgr2=0x7f23b806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:03.201 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 0x7f23b806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 >> v2:172.21.15.121:3300/0 conn(0x7f23b80c4ae0 msgr2=0x7f23b806c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 --2- 172.21.15.90:0/1475941001 >> v2:172.21.15.121:3300/0 conn(0x7f23b80c4ae0 0x7f23b806c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b806cea0 con 0x7f23b8149620 2023-03-03T06:18:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f23b4001670 con 0x7f23b8149620 2023-03-03T06:18:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 >> v1:172.21.15.90:6789/0 conn(0x7f23b8149620 legacy=0x7f23b80c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 0x7f23a4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23a4003f80 0x7f23a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23a4004d90 0x7f23a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b806b890 con 0x7f23a4004d90 2023-03-03T06:18:03.204 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b814fbb0 con 0x7f23a4003f80 2023-03-03T06:18:03.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a400ba10 con 0x7f23b80c3f60 2023-03-03T06:18:03.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.189+0000 7f23bebfb700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23a4004d90 0x7f23a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.206 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23be3fa700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 0x7f23a4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bdbf9700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23a4003f80 0x7f23a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bebfb700 1 -- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 msgr2=0x7f23a4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:03.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bebfb700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23b80c3f60 0x7f23a4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bebfb700 1 -- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23a4003f80 msgr2=0x7f23a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:03.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bebfb700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23a4003f80 0x7f23a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bebfb700 1 -- 172.21.15.90:0/1475941001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40035b0 con 0x7f23a4004d90 2023-03-03T06:18:03.207 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bebfb700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23a4004d90 0x7f23a4005200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009fc0 tx=0x7f23a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5eb510b6087707 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:03.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b000ec30 con 0x7f23a4004d90 2023-03-03T06:18:03.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b000f040 con 0x7f23a4004d90 2023-03-03T06:18:03.208 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.190+0000 7f23bd3f8700 1 -- 172.21.15.90:0/1475941001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0014480 con 0x7f23a4004d90 2023-03-03T06:18:03.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.191+0000 7f23c065e700 1 -- 172.21.15.90:0/1475941001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23a4004d90 msgr2=0x7f23a4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:03.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.191+0000 7f23c065e700 1 --2- 172.21.15.90:0/1475941001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23a4004d90 0x7f23a4005200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009fc0 tx=0x7f23a4005880 comp rx=0 tx=0).stop 2023-03-03T06:18:03.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.191+0000 7f23bebfb700 1 -- 172.21.15.90:0/1475941001 reap_dead start 2023-03-03T06:18:03.209 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.191+0000 7f23c065e700 1 -- 172.21.15.90:0/1475941001 shutdown_connections 2023-03-03T06:18:03.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.191+0000 7f23c065e700 1 -- 172.21.15.90:0/1475941001 >> 172.21.15.90:0/1475941001 conn(0x7f23b80a7060 msgr2=0x7f23b80a7460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:03.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.191+0000 7f23c065e700 1 -- 172.21.15.90:0/1475941001 shutdown_connections 2023-03-03T06:18:03.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.191+0000 7f23c065e700 1 -- 172.21.15.90:0/1475941001 wait complete. 2023-03-03T06:18:03.210 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 Processor -- start 2023-03-03T06:18:03.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 -- start start 2023-03-03T06:18:03.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23a4004d90 0x7f23b81da910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.211 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b81e4d00 0x7f23b81e5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b814a690 0x7f23b814ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b814c220 con 0x7f23b814a690 2023-03-03T06:18:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b814c5a0 con 0x7f23b81e4d00 2023-03-03T06:18:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.192+0000 7f23c065e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8149eb0 con 0x7f23a4004d90 2023-03-03T06:18:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b814a690 0x7f23b814ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b814a690 0x7f23b814ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33868/0 (socket says 172.21.15.90:33868) 2023-03-03T06:18:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 -- 172.21.15.90:0/3198716127 learned_addr learned my addr 172.21.15.90:0/3198716127 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bdbf9700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b81e4d00 0x7f23b81e5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23be3fa700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23a4004d90 0x7f23b81da910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 -- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23a4004d90 msgr2=0x7f23b81da910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23a4004d90 0x7f23b81da910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 -- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b81e4d00 msgr2=0x7f23b81e5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:03.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b81e4d00 0x7f23b81e5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:03.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 -- 172.21.15.90:0/3198716127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b81500a0 con 0x7f23b814a690 2023-03-03T06:18:03.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23bebfb700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b814a690 0x7f23b814ab00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23b0016000 tx=0x7f23b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:03.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23aeffd700 1 -- 172.21.15.90:0/3198716127 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0007000 con 0x7f23b814a690 2023-03-03T06:18:03.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23aeffd700 1 -- 172.21.15.90:0/3198716127 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b0015a80 con 0x7f23b814a690 2023-03-03T06:18:03.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b80bffd0 con 0x7f23b814a690 2023-03-03T06:18:03.214 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23aeffd700 1 -- 172.21.15.90:0/3198716127 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0015e70 con 0x7f23b814a690 2023-03-03T06:18:03.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.193+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b814fdc0 con 0x7f23b814a690 2023-03-03T06:18:03.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.194+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b814ee20 con 0x7f23b814a690 2023-03-03T06:18:03.215 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.194+0000 7f23aeffd700 1 -- 172.21.15.90:0/3198716127 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f23b002d030 con 0x7f23b814a690 2023-03-03T06:18:03.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.195+0000 7f23aeffd700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f239c0682f0 0x7f239c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:03.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.195+0000 7f23be3fa700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f239c0682f0 0x7f239c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:03.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.195+0000 7f23aeffd700 1 -- 172.21.15.90:0/3198716127 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f23b0086f20 con 0x7f23b814a690 2023-03-03T06:18:03.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.195+0000 7f23be3fa700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f239c0682f0 0x7f239c06a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f23b4001cf0 tx=0x7f23b4007980 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:03.216 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.199+0000 7f23aeffd700 1 -- 172.21.15.90:0/3198716127 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23b0053ad0 con 0x7f23b814a690 2023-03-03T06:18:03.384 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:03.382+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f23b804e7c0 con 0x7f23b814a690 2023-03-03T06:18:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.351+0000 7f23aeffd700 1 -- 172.21.15.90:0/3198716127 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f23b00570f0 con 0x7f23b814a690 2023-03-03T06:18:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f239c0682f0 msgr2=0x7f239c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f239c0682f0 0x7f239c06a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f23b4001cf0 tx=0x7f23b4007980 comp rx=0 tx=0).stop 2023-03-03T06:18:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b814a690 msgr2=0x7f23b814ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b814a690 0x7f23b814ab00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23b0016000 tx=0x7f23b0009750 comp rx=0 tx=0).stop 2023-03-03T06:18:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 shutdown_connections 2023-03-03T06:18:04.357 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f239c0682f0 0x7f239c06a7a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.357 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b814a690 0x7f23b814ab00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.357 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23a4004d90 0x7f23b81da910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.357 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 --2- 172.21.15.90:0/3198716127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b81e4d00 0x7f23b81e5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.357 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 >> 172.21.15.90:0/3198716127 conn(0x7f23b80a7060 msgr2=0x7f23b806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:04.357 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 shutdown_connections 2023-03-03T06:18:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.354+0000 7f23c065e700 1 -- 172.21.15.90:0/3198716127 wait complete. 2023-03-03T06:18:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2023-03-03T06:18:04.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T06:18:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.502+0000 7fed5d41b700 1 Processor -- start 2023-03-03T06:18:04.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.503+0000 7fed5d41b700 1 -- start start 2023-03-03T06:18:04.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.503+0000 7fed5d41b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 0x7fed580c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.503+0000 7fed5d41b700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed580729e0 con 0x7fed5806d920 2023-03-03T06:18:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.503+0000 7fed5d41b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed580bde70 con 0x7fed5806e1c0 2023-03-03T06:18:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.503+0000 7fed5d41b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed580707f0 con 0x7fed580c2810 2023-03-03T06:18:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.503+0000 7fed567fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fed5806d920 0x7fed581504d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53080/0 (socket says 172.21.15.90:53080) 2023-03-03T06:18:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.503+0000 7fed567fc700 1 -- 172.21.15.90:0/4271017036 learned_addr learned my addr 172.21.15.90:0/4271017036 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed56ffd700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 0x7fed580c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392305146 0 0) 0x7fed580729e0 con 0x7fed5806d920 2023-03-03T06:18:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed400035b0 con 0x7fed5806d920 2023-03-03T06:18:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351934595 0 0) 0x7fed580bde70 con 0x7fed5806e1c0 2023-03-03T06:18:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed580729e0 con 0x7fed5806e1c0 2023-03-03T06:18:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15276876 0 0) 0x7fed400035b0 con 0x7fed5806d920 2023-03-03T06:18:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 msgr2=0x7fed580c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 0x7fed580c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 >> v1:172.21.15.121:6789/0 conn(0x7fed5806e1c0 legacy=0x7fed58157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58072440 con 0x7fed5806d920 2023-03-03T06:18:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7fed48001670 con 0x7fed5806d920 2023-03-03T06:18:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 >> v1:172.21.15.90:6789/0 conn(0x7fed5806d920 legacy=0x7fed581504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 0x7fed40003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed40004170 0x7fed40004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed40004d90 0x7fed40005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4000bba0 con 0x7fed40004d90 2023-03-03T06:18:04.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4000bce0 con 0x7fed40004170 2023-03-03T06:18:04.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4000bea0 con 0x7fed580c2810 2023-03-03T06:18:04.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed577fe700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed40004d90 0x7fed40005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed56ffd700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 0x7fed40003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed567fc700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed40004170 0x7fed40004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed577fe700 1 -- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 msgr2=0x7fed40003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:04.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed577fe700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed580c2810 0x7fed40003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed577fe700 1 -- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed40004170 msgr2=0x7fed40004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:04.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed577fe700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed40004170 0x7fed40004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed577fe700 1 -- 172.21.15.90:0/4271017036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400035b0 con 0x7fed40004d90 2023-03-03T06:18:04.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.504+0000 7fed577fe700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed40004d90 0x7fed40005200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fed5000c490 tx=0x7fed40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ed55aeb575b4de server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:04.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed50003a00 con 0x7fed40004d90 2023-03-03T06:18:04.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed50003b60 con 0x7fed40004d90 2023-03-03T06:18:04.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed55ffb700 1 -- 172.21.15.90:0/4271017036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed500050d0 con 0x7fed40004d90 2023-03-03T06:18:04.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed5d41b700 1 -- 172.21.15.90:0/4271017036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed40004d90 msgr2=0x7fed40005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:04.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed5d41b700 1 --2- 172.21.15.90:0/4271017036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed40004d90 0x7fed40005200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fed5000c490 tx=0x7fed40005880 comp rx=0 tx=0).stop 2023-03-03T06:18:04.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed577fe700 1 -- 172.21.15.90:0/4271017036 reap_dead start 2023-03-03T06:18:04.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed5d41b700 1 -- 172.21.15.90:0/4271017036 shutdown_connections 2023-03-03T06:18:04.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed5d41b700 1 -- 172.21.15.90:0/4271017036 >> 172.21.15.90:0/4271017036 conn(0x7fed5806c0d0 msgr2=0x7fed580c2100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:04.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed5d41b700 1 -- 172.21.15.90:0/4271017036 shutdown_connections 2023-03-03T06:18:04.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.505+0000 7fed5d41b700 1 -- 172.21.15.90:0/4271017036 wait complete. 2023-03-03T06:18:04.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.506+0000 7fed5d41b700 1 Processor -- start 2023-03-03T06:18:04.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.506+0000 7fed5d41b700 1 -- start start 2023-03-03T06:18:04.522 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed5d41b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed581458b0 0x7fed581e3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.522 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed5d41b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed581438a0 0x7fed581e4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed5d41b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed581449f0 0x7fed581e5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed5d41b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed581e4a40 con 0x7fed581458b0 2023-03-03T06:18:04.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed5d41b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed581e3b90 con 0x7fed581438a0 2023-03-03T06:18:04.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed5d41b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed58143d30 con 0x7fed581449f0 2023-03-03T06:18:04.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed581458b0 0x7fed581e3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed581458b0 0x7fed581e3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33886/0 (socket says 172.21.15.90:33886) 2023-03-03T06:18:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 -- 172.21.15.90:0/3264157896 learned_addr learned my addr 172.21.15.90:0/3264157896 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed577fe700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed581449f0 0x7fed581e5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed567fc700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed581438a0 0x7fed581e4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 -- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed581449f0 msgr2=0x7fed581e5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed581449f0 0x7fed581e5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 -- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed581438a0 msgr2=0x7fed581e4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed581438a0 0x7fed581e4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.507+0000 7fed56ffd700 1 -- 172.21.15.90:0/3264157896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5806e920 con 0x7fed581458b0 2023-03-03T06:18:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.508+0000 7fed56ffd700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed581458b0 0x7fed581e3fc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed4c011aa0 tx=0x7fed4c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.508+0000 7fed3f7fe700 1 -- 172.21.15.90:0/3264157896 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c011ee0 con 0x7fed581458b0 2023-03-03T06:18:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.508+0000 7fed3f7fe700 1 -- 172.21.15.90:0/3264157896 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c00f940 con 0x7fed581458b0 2023-03-03T06:18:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.508+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5806dd40 con 0x7fed581458b0 2023-03-03T06:18:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.508+0000 7fed3f7fe700 1 -- 172.21.15.90:0/3264157896 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00fcf0 con 0x7fed581458b0 2023-03-03T06:18:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.508+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed58072700 con 0x7fed581458b0 2023-03-03T06:18:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.509+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed580b9900 con 0x7fed581458b0 2023-03-03T06:18:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.509+0000 7fed3f7fe700 1 -- 172.21.15.90:0/3264157896 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed4c014440 con 0x7fed581458b0 2023-03-03T06:18:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.509+0000 7fed3f7fe700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fed440682f0 0x7fed4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.510+0000 7fed3f7fe700 1 -- 172.21.15.90:0/3264157896 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fed4c086340 con 0x7fed581458b0 2023-03-03T06:18:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.510+0000 7fed567fc700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fed440682f0 0x7fed4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.510+0000 7fed567fc700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fed440682f0 0x7fed4406a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed580bf970 tx=0x7fed48000fc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.514+0000 7fed3f7fe700 1 -- 172.21.15.90:0/3264157896 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed4c052cf0 con 0x7fed581458b0 2023-03-03T06:18:04.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:18:04.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:18:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:04.700+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fed5804e750 con 0x7fed581458b0 2023-03-03T06:18:05.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.357+0000 7fed3f7fe700 1 -- 172.21.15.90:0/3264157896 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) v1 ==== 170+0+0 (secure 0 0 0) 0x7fed4c056310 con 0x7fed581458b0 2023-03-03T06:18:05.364 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.363+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fed440682f0 msgr2=0x7fed4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.363+0000 7fed5d41b700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fed440682f0 0x7fed4406a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed580bf970 tx=0x7fed48000fc0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.363+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed581458b0 msgr2=0x7fed581e3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.363+0000 7fed5d41b700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed581458b0 0x7fed581e3fc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed4c011aa0 tx=0x7fed4c011db0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.366 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.364+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 shutdown_connections 2023-03-03T06:18:05.366 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.364+0000 7fed5d41b700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed581449f0 0x7fed581e5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.366 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.364+0000 7fed5d41b700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fed440682f0 0x7fed4406a7a0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed580bf970 tx=0x7fed48000fc0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.366 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.364+0000 7fed5d41b700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed581458b0 0x7fed581e3fc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.364+0000 7fed5d41b700 1 --2- 172.21.15.90:0/3264157896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed581438a0 0x7fed581e4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.364+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 >> 172.21.15.90:0/3264157896 conn(0x7fed5806c0d0 msgr2=0x7fed580b91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.365+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 shutdown_connections 2023-03-03T06:18:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.366+0000 7fed5d41b700 1 -- 172.21.15.90:0/3264157896 wait complete. 2023-03-03T06:18:05.370 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2023-03-03T06:18:05.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-03T06:18:05.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.515+0000 7f12c4216700 1 Processor -- start 2023-03-03T06:18:05.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.515+0000 7f12c4216700 1 -- start start 2023-03-03T06:18:05.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.516+0000 7f12c4216700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12bc0bddc0 0x7f12bc0b9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.516+0000 7f12c4216700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc157810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.516+0000 7f12c4216700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc0ba050 con 0x7f12bc147030 2023-03-03T06:18:05.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.516+0000 7f12c4216700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc0b8820 con 0x7f12bc0bddc0 2023-03-03T06:18:05.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.516+0000 7f12c4216700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc14deb0 con 0x7f12bc0b9bd0 2023-03-03T06:18:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c17b1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12bc0bddc0 0x7f12bc0b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c27b3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c1fb2700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f12bc147030 0x7f12bc14d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:45742/0 (socket says 172.21.15.90:45742) 2023-03-03T06:18:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c27b3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:38832/0 (socket says 172.21.15.90:38832) 2023-03-03T06:18:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c17b1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12bc0bddc0 0x7f12bc0b9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48036/0 (socket says 172.21.15.90:48036) 2023-03-03T06:18:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c1fb2700 1 -- 172.21.15.90:0/3265353073 learned_addr learned my addr 172.21.15.90:0/3265353073 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c17b1700 1 -- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 msgr2=0x7f12bc157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c17b1700 1 --2- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c17b1700 1 -- 172.21.15.90:0/3265353073 >> v1:172.21.15.121:6789/0 conn(0x7f12bc147030 legacy=0x7f12bc14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c0fb0700 1 -- 172.21.15.90:0/3265353073 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505920999 0 0) 0x7f12bc0ba050 con 0x7f12bc147030 2023-03-03T06:18:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c17b1700 1 -- 172.21.15.90:0/3265353073 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc072500 con 0x7f12bc0bddc0 2023-03-03T06:18:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c27b3700 1 --2- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c17b1700 1 --2- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 0x7f12bc0b9690 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009870 tx=0x7f12b8009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1223235cb9412424 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c0fb0700 1 -- 172.21.15.90:0/3265353073 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8011830 con 0x7f12bc0bddc0 2023-03-03T06:18:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c0fb0700 1 -- 172.21.15.90:0/3265353073 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b8011990 con 0x7f12bc0bddc0 2023-03-03T06:18:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.517+0000 7f12c0fb0700 1 -- 172.21.15.90:0/3265353073 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8011d20 con 0x7f12bc0bddc0 2023-03-03T06:18:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 -- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 msgr2=0x7f12bc0b9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 --2- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 0x7f12bc0b9690 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009870 tx=0x7f12b8009b80 comp rx=0 tx=0).stop 2023-03-03T06:18:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 -- 172.21.15.90:0/3265353073 shutdown_connections 2023-03-03T06:18:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 --2- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 --2- 172.21.15.90:0/3265353073 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 0x7f12bc0b9690 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 -- 172.21.15.90:0/3265353073 >> 172.21.15.90:0/3265353073 conn(0x7f12bc0c3230 msgr2=0x7f12bc0c3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 -- 172.21.15.90:0/3265353073 shutdown_connections 2023-03-03T06:18:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.518+0000 7f12c4216700 1 -- 172.21.15.90:0/3265353073 wait complete. 2023-03-03T06:18:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.519+0000 7f12c4216700 1 Processor -- start 2023-03-03T06:18:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.519+0000 7f12c4216700 1 -- start start 2023-03-03T06:18:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc1e3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 0x7f12bc0c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12bc147030 0x7f12bc1e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc1e2bf0 con 0x7f12bc147030 2023-03-03T06:18:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc1e5ce0 con 0x7f12bc0bddc0 2023-03-03T06:18:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc1e5fa0 con 0x7f12bc0b9bd0 2023-03-03T06:18:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12bc147030 0x7f12bc1e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12bc147030 0x7f12bc1e52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33902/0 (socket says 172.21.15.90:33902) 2023-03-03T06:18:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 -- 172.21.15.90:0/3779279761 learned_addr learned my addr 172.21.15.90:0/3779279761 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c1fb2700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc1e3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c17b1700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 0x7f12bc0c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 -- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 msgr2=0x7f12bc1e3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc1e3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 -- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 msgr2=0x7f12bc0c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 0x7f12bc0c23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 -- 172.21.15.90:0/3779279761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009a70 con 0x7f12bc147030 2023-03-03T06:18:05.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c27b3700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12bc147030 0x7f12bc1e52c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12b400efc0 tx=0x7f12b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:05.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12b27fc700 1 -- 172.21.15.90:0/3779279761 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400ca90 con 0x7f12bc147030 2023-03-03T06:18:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12b27fc700 1 -- 172.21.15.90:0/3779279761 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b400f040 con 0x7f12bc147030 2023-03-03T06:18:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b80094a0 con 0x7f12bc147030 2023-03-03T06:18:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12b27fc700 1 -- 172.21.15.90:0/3779279761 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400ce10 con 0x7f12bc147030 2023-03-03T06:18:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.520+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc072220 con 0x7f12bc147030 2023-03-03T06:18:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.521+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc0ba280 con 0x7f12bc147030 2023-03-03T06:18:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.522+0000 7f12b27fc700 1 -- 172.21.15.90:0/3779279761 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f12b4014480 con 0x7f12bc147030 2023-03-03T06:18:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.522+0000 7f12b27fc700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f12a0068320 0x7f12a006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.522+0000 7f12b27fc700 1 -- 172.21.15.90:0/3779279761 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f12b40864b0 con 0x7f12bc147030 2023-03-03T06:18:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.522+0000 7f12c1fb2700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f12a0068320 0x7f12a006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.523+0000 7f12c1fb2700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f12a0068320 0x7f12a006a7d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f12ac000f60 tx=0x7f12ac007e70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.526+0000 7f12b27fc700 1 -- 172.21.15.90:0/3779279761 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12b4052c20 con 0x7f12bc147030 2023-03-03T06:18:05.752 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.751+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f12bc0b8f60 con 0x7f12bc147030 2023-03-03T06:18:05.759 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.757+0000 7f12b27fc700 1 -- 172.21.15.90:0/3779279761 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f12b4009d70 con 0x7f12bc147030 2023-03-03T06:18:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f12a0068320 msgr2=0x7f12a006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f12a0068320 0x7f12a006a7d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f12ac000f60 tx=0x7f12ac007e70 comp rx=0 tx=0).stop 2023-03-03T06:18:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12bc147030 msgr2=0x7f12bc1e52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12bc147030 0x7f12bc1e52c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12b400efc0 tx=0x7f12b400c3a0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 shutdown_connections 2023-03-03T06:18:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f12a0068320 0x7f12a006a7d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12bc147030 0x7f12bc1e52c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12bc0b9bd0 0x7f12bc1e3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 --2- 172.21.15.90:0/3779279761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12bc0bddc0 0x7f12bc0c23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 >> 172.21.15.90:0/3779279761 conn(0x7f12bc0c3230 msgr2=0x7f12bc1468a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 shutdown_connections 2023-03-03T06:18:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.762+0000 7f12c4216700 1 -- 172.21.15.90:0/3779279761 wait complete. 2023-03-03T06:18:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 3 and data pool 4 2023-03-03T06:18:05.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:05.906 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f0229469700 1 Processor -- start 2023-03-03T06:18:05.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f0229469700 1 -- start start 2023-03-03T06:18:05.908 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f0229469700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f022406a610 0x7f022406aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f0229469700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02240c08a0 con 0x7f0224157b80 2023-03-03T06:18:05.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f0229469700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02240c1d50 con 0x7f0224149760 2023-03-03T06:18:05.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f0229469700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022406b970 con 0x7f022406a610 2023-03-03T06:18:05.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f02237fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0224157b80 0x7f0224159f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53128/0 (socket says 172.21.15.90:53128) 2023-03-03T06:18:05.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.905+0000 7f02237fe700 1 -- 172.21.15.90:0/2607495417 learned_addr learned my addr 172.21.15.90:0/2607495417 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:05.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f02227fc700 1 --2- 172.21.15.90:0/2607495417 >> v2:172.21.15.121:3300/0 conn(0x7f022406a610 0x7f022406aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f0221ffb700 1 -- 172.21.15.90:0/2607495417 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776569873 0 0) 0x7f02240c08a0 con 0x7f0224157b80 2023-03-03T06:18:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f0221ffb700 1 -- 172.21.15.90:0/2607495417 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c0035b0 con 0x7f0224157b80 2023-03-03T06:18:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f0221ffb700 1 -- 172.21.15.90:0/2607495417 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566524611 0 0) 0x7f020c0035b0 con 0x7f0224157b80 2023-03-03T06:18:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f02227fc700 1 -- 172.21.15.90:0/2607495417 >> v1:172.21.15.121:6789/0 conn(0x7f0224149760 legacy=0x7f02240c2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f02227fc700 1 -- 172.21.15.90:0/2607495417 >> v1:172.21.15.90:6789/0 conn(0x7f0224157b80 legacy=0x7f0224159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f02227fc700 1 -- 172.21.15.90:0/2607495417 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022408b9c0 con 0x7f022406a610 2023-03-03T06:18:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f02227fc700 1 --2- 172.21.15.90:0/2607495417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f022406aab0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0218009870 tx=0x7f0218009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=198550c218dcf9e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f0221ffb700 1 -- 172.21.15.90:0/2607495417 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c0019f0 con 0x7f022406a610 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f0221ffb700 1 -- 172.21.15.90:0/2607495417 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218004030 con 0x7f022406a610 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.906+0000 7f0221ffb700 1 -- 172.21.15.90:0/2607495417 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02180043a0 con 0x7f022406a610 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.907+0000 7f0229469700 1 -- 172.21.15.90:0/2607495417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 msgr2=0x7f022406aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.907+0000 7f0229469700 1 --2- 172.21.15.90:0/2607495417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f022406aab0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0218009870 tx=0x7f0218009b80 comp rx=0 tx=0).stop 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.907+0000 7f0229469700 1 -- 172.21.15.90:0/2607495417 shutdown_connections 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.907+0000 7f0229469700 1 --2- 172.21.15.90:0/2607495417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f022406aab0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.907+0000 7f0229469700 1 -- 172.21.15.90:0/2607495417 >> 172.21.15.90:0/2607495417 conn(0x7f02240c6160 msgr2=0x7f02240c1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:05.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.907+0000 7f0229469700 1 -- 172.21.15.90:0/2607495417 shutdown_connections 2023-03-03T06:18:05.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.907+0000 7f0229469700 1 -- 172.21.15.90:0/2607495417 wait complete. 2023-03-03T06:18:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.908+0000 7f0229469700 1 Processor -- start 2023-03-03T06:18:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.908+0000 7f0229469700 1 -- start start 2023-03-03T06:18:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0229469700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f02240b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0229469700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0224149760 0x7f02241e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0229469700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0224157b80 0x7f0224148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0229469700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02241e41a0 con 0x7f0224149760 2023-03-03T06:18:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0229469700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02240b8720 con 0x7f022406a610 2023-03-03T06:18:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0229469700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02241e8610 con 0x7f0224157b80 2023-03-03T06:18:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0224149760 0x7f02241e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0224149760 0x7f02241e80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33926/0 (socket says 172.21.15.90:33926) 2023-03-03T06:18:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 -- 172.21.15.90:0/227382036 learned_addr learned my addr 172.21.15.90:0/227382036 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02237fe700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0224157b80 0x7f0224148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0222ffd700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f02240b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 -- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0224157b80 msgr2=0x7f0224148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0224157b80 0x7f0224148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 -- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 msgr2=0x7f02240b81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:05.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f02240b81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:05.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 -- 172.21.15.90:0/227382036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180094a0 con 0x7f0224149760 2023-03-03T06:18:05.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0222ffd700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f02240b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:05.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f02227fc700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0224149760 0x7f02241e80d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f022408b8e0 tx=0x7f0218004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:05.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f020b7fe700 1 -- 172.21.15.90:0/227382036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218010590 con 0x7f0224149760 2023-03-03T06:18:05.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f020b7fe700 1 -- 172.21.15.90:0/227382036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02180106f0 con 0x7f0224149760 2023-03-03T06:18:05.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02240c19e0 con 0x7f0224149760 2023-03-03T06:18:05.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.909+0000 7f020b7fe700 1 -- 172.21.15.90:0/227382036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218010aa0 con 0x7f0224149760 2023-03-03T06:18:05.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.910+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02240c01f0 con 0x7f0224149760 2023-03-03T06:18:05.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.910+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02240c2d70 con 0x7f0224149760 2023-03-03T06:18:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.911+0000 7f020b7fe700 1 -- 172.21.15.90:0/227382036 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f021800f050 con 0x7f0224149760 2023-03-03T06:18:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.911+0000 7f020b7fe700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0210068320 0x7f021006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.911+0000 7f020b7fe700 1 -- 172.21.15.90:0/227382036 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0218087220 con 0x7f0224149760 2023-03-03T06:18:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.911+0000 7f0222ffd700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0210068320 0x7f021006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.911+0000 7f0222ffd700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0210068320 0x7f021006a7d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0214000f60 tx=0x7f0214009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:05.915+0000 7f020b7fe700 1 -- 172.21.15.90:0/227382036 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0218053a80 con 0x7f0224149760 2023-03-03T06:18:06.100 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.098+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f022404e750 con 0x7f0224149760 2023-03-03T06:18:06.101 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.099+0000 7f020b7fe700 1 -- 172.21.15.90:0/227382036 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17824 (secure 0 0 0) 0x7f02180570a0 con 0x7f0224149760 2023-03-03T06:18:06.101 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:06.102 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":22,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:05.752298+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T06:18:03.400568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T06:18:04.718088+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0210068320 msgr2=0x7f021006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0210068320 0x7f021006a7d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0214000f60 tx=0x7f0214009040 comp rx=0 tx=0).stop 2023-03-03T06:18:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0224149760 msgr2=0x7f02241e80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0224149760 0x7f02241e80d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f022408b8e0 tx=0x7f0218004750 comp rx=0 tx=0).stop 2023-03-03T06:18:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 shutdown_connections 2023-03-03T06:18:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0224157b80 0x7f0224148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022406a610 0x7f02240b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0210068320 0x7f021006a7d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 --2- 172.21.15.90:0/227382036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0224149760 0x7f02241e80d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 >> 172.21.15.90:0/227382036 conn(0x7f02240c6160 msgr2=0x7f02240a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 shutdown_connections 2023-03-03T06:18:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.101+0000 7f0229469700 1 -- 172.21.15.90:0/227382036 wait complete. 2023-03-03T06:18:06.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:06.236 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.235+0000 7fdd03526700 1 Processor -- start 2023-03-03T06:18:06.237 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.235+0000 7fdd03526700 1 -- start start 2023-03-03T06:18:06.238 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd03526700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0c2860 0x7fdcfc0c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd03526700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdcfc14fe60 0x7fdcfc156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd03526700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc146310 con 0x7fdcfc0bd220 2023-03-03T06:18:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd03526700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcfc06d420 con 0x7fdcfc14fe60 2023-03-03T06:18:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd03526700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc14cc50 con 0x7fdcfc0c2860 2023-03-03T06:18:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdcfc14fe60 0x7fdcfc156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdcfc14fe60 0x7fdcfc156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33936/0 (socket says 172.21.15.90:33936) 2023-03-03T06:18:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 -- 172.21.15.90:0/3050023541 learned_addr learned my addr 172.21.15.90:0/3050023541 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd012c2700 1 --2- 172.21.15.90:0/3050023541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0c2860 0x7fdcfc0c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 -- 172.21.15.90:0/3050023541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0c2860 msgr2=0x7fdcfc0c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 --2- 172.21.15.90:0/3050023541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0c2860 0x7fdcfc0c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 -- 172.21.15.90:0/3050023541 >> v1:172.21.15.121:6789/0 conn(0x7fdcfc0bd220 legacy=0x7fdcfc14f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 -- 172.21.15.90:0/3050023541 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc0bd770 con 0x7fdcfc14fe60 2023-03-03T06:18:06.240 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdd01ac3700 1 --2- 172.21.15.90:0/3050023541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc156770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009e50 tx=0x7fdcf400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630ca7599b4cda37 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdcf3fff700 1 -- 172.21.15.90:0/3050023541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf400e070 con 0x7fdcfc14fe60 2023-03-03T06:18:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdcf3fff700 1 -- 172.21.15.90:0/3050023541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf4003a20 con 0x7fdcfc14fe60 2023-03-03T06:18:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.236+0000 7fdcf3fff700 1 -- 172.21.15.90:0/3050023541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4003dd0 con 0x7fdcfc14fe60 2023-03-03T06:18:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 -- 172.21.15.90:0/3050023541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc14fe60 msgr2=0x7fdcfc156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 --2- 172.21.15.90:0/3050023541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc156770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009e50 tx=0x7fdcf400b330 comp rx=0 tx=0).stop 2023-03-03T06:18:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 -- 172.21.15.90:0/3050023541 shutdown_connections 2023-03-03T06:18:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 --2- 172.21.15.90:0/3050023541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc156770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 --2- 172.21.15.90:0/3050023541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0c2860 0x7fdcfc0c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 -- 172.21.15.90:0/3050023541 >> 172.21.15.90:0/3050023541 conn(0x7fdcfc06c0d0 msgr2=0x7fdcfc0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 -- 172.21.15.90:0/3050023541 shutdown_connections 2023-03-03T06:18:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.237+0000 7fdd03526700 1 -- 172.21.15.90:0/3050023541 wait complete. 2023-03-03T06:18:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.238+0000 7fdd03526700 1 Processor -- start 2023-03-03T06:18:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.238+0000 7fdd03526700 1 -- start start 2023-03-03T06:18:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd03526700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0bd220 0x7fdcfc06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd03526700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc0c2860 0x7fdcfc13bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd03526700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc1d7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd03526700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc13b880 con 0x7fdcfc0c2860 2023-03-03T06:18:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd03526700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc1e2920 con 0x7fdcfc14fe60 2023-03-03T06:18:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd03526700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc1e38f0 con 0x7fdcfc0bd220 2023-03-03T06:18:06.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc0c2860 0x7fdcfc13bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.246 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc0c2860 0x7fdcfc13bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33940/0 (socket says 172.21.15.90:33940) 2023-03-03T06:18:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 -- 172.21.15.90:0/1763656651 learned_addr learned my addr 172.21.15.90:0/1763656651 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd012c2700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0bd220 0x7fdcfc06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd01ac3700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc1d7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 -- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0bd220 msgr2=0x7fdcfc06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0bd220 0x7fdcfc06a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 -- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcfc14fe60 msgr2=0x7fdcfc1d7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc1d7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 -- 172.21.15.90:0/1763656651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf4009b30 con 0x7fdcfc0c2860 2023-03-03T06:18:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd01ac3700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc1d7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd00ac1700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc0c2860 0x7fdcfc13bef0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00b4d0 tx=0x7fdcec00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdcf1ffb700 1 -- 172.21.15.90:0/1763656651 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec011830 con 0x7fdcfc0c2860 2023-03-03T06:18:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdcf1ffb700 1 -- 172.21.15.90:0/1763656651 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcec011990 con 0x7fdcfc0c2860 2023-03-03T06:18:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc0bdd30 con 0x7fdcfc0c2860 2023-03-03T06:18:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.239+0000 7fdcf1ffb700 1 -- 172.21.15.90:0/1763656651 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec011d40 con 0x7fdcfc0c2860 2023-03-03T06:18:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.240+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc0bda50 con 0x7fdcfc0c2860 2023-03-03T06:18:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.241+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc0723b0 con 0x7fdcfc0c2860 2023-03-03T06:18:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.241+0000 7fdcf1ffb700 1 -- 172.21.15.90:0/1763656651 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdcec00f050 con 0x7fdcfc0c2860 2023-03-03T06:18:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.241+0000 7fdcf1ffb700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fdce0068320 0x7fdce006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.250 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.241+0000 7fdd012c2700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fdce0068320 0x7fdce006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.250 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.241+0000 7fdcf1ffb700 1 -- 172.21.15.90:0/1763656651 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdcec085820 con 0x7fdcfc0c2860 2023-03-03T06:18:06.250 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.242+0000 7fdd012c2700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fdce0068320 0x7fdce006a7d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8006b20 tx=0x7fdcf8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:06.250 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.245+0000 7fdcf1ffb700 1 -- 172.21.15.90:0/1763656651 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdcec051f90 con 0x7fdcfc0c2860 2023-03-03T06:18:06.431 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.430+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcfc06aae0 con 0x7fdcfc0c2860 2023-03-03T06:18:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.431+0000 7fdcf1ffb700 1 -- 172.21.15.90:0/1763656651 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17824 (secure 0 0 0) 0x7fdcec015020 con 0x7fdcfc0c2860 2023-03-03T06:18:06.433 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:06.433 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":22,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:05.752298+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T06:18:03.400568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T06:18:04.718088+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fdce0068320 msgr2=0x7fdce006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fdce0068320 0x7fdce006a7d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8006b20 tx=0x7fdcf8008040 comp rx=0 tx=0).stop 2023-03-03T06:18:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc0c2860 msgr2=0x7fdcfc13bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc0c2860 0x7fdcfc13bef0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00b4d0 tx=0x7fdcec00b7e0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.435 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 shutdown_connections 2023-03-03T06:18:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcfc14fe60 0x7fdcfc1d7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcfc0bd220 0x7fdcfc06a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fdce0068320 0x7fdce006a7d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 --2- 172.21.15.90:0/1763656651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdcfc0c2860 0x7fdcfc13bef0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 >> 172.21.15.90:0/1763656651 conn(0x7fdcfc06c0d0 msgr2=0x7fdcfc13e950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 shutdown_connections 2023-03-03T06:18:06.436 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.433+0000 7fdd03526700 1 -- 172.21.15.90:0/1763656651 wait complete. 2023-03-03T06:18:06.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-03T06:18:06.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.569+0000 7f74df48f700 1 Processor -- start 2023-03-03T06:18:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.569+0000 7f74df48f700 1 -- start start 2023-03-03T06:18:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74df48f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f74d80c2860 0x7f74d80c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74df48f700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f74d814fe60 0x7f74d8156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74df48f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d8146310 con 0x7f74d80bd220 2023-03-03T06:18:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74df48f700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d806d420 con 0x7f74d814fe60 2023-03-03T06:18:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74df48f700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d814cc50 con 0x7f74d80c2860 2023-03-03T06:18:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f74d814fe60 0x7f74d8156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f74d814fe60 0x7f74d8156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33950/0 (socket says 172.21.15.90:33950) 2023-03-03T06:18:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 -- 172.21.15.90:0/3222598664 learned_addr learned my addr 172.21.15.90:0/3222598664 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 -- 172.21.15.90:0/3222598664 >> v2:172.21.15.121:3300/0 conn(0x7f74d80c2860 msgr2=0x7f74d80c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dd22b700 1 --2- 172.21.15.90:0/3222598664 >> v2:172.21.15.121:3300/0 conn(0x7f74d80c2860 0x7f74d80c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 --2- 172.21.15.90:0/3222598664 >> v2:172.21.15.121:3300/0 conn(0x7f74d80c2860 0x7f74d80c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 -- 172.21.15.90:0/3222598664 >> v1:172.21.15.121:6789/0 conn(0x7f74d80bd220 legacy=0x7f74d814f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 -- 172.21.15.90:0/3222598664 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d80bd770 con 0x7f74d814fe60 2023-03-03T06:18:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dd22b700 1 --2- 172.21.15.90:0/3222598664 >> v2:172.21.15.121:3300/0 conn(0x7f74d80c2860 0x7f74d80c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74dda2c700 1 --2- 172.21.15.90:0/3222598664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d814fe60 0x7f74d8156770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f74d80676e0 tx=0x7f74d000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d95d8cc9b5416f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:06.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74cffff700 1 -- 172.21.15.90:0/3222598664 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0010070 con 0x7f74d814fe60 2023-03-03T06:18:06.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74cffff700 1 -- 172.21.15.90:0/3222598664 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d000dd50 con 0x7f74d814fe60 2023-03-03T06:18:06.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.570+0000 7f74cffff700 1 -- 172.21.15.90:0/3222598664 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d000b310 con 0x7f74d814fe60 2023-03-03T06:18:06.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 -- 172.21.15.90:0/3222598664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d814fe60 msgr2=0x7f74d8156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 --2- 172.21.15.90:0/3222598664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d814fe60 0x7f74d8156770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f74d80676e0 tx=0x7f74d000d430 comp rx=0 tx=0).stop 2023-03-03T06:18:06.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 -- 172.21.15.90:0/3222598664 shutdown_connections 2023-03-03T06:18:06.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 --2- 172.21.15.90:0/3222598664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d814fe60 0x7f74d8156770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 --2- 172.21.15.90:0/3222598664 >> v2:172.21.15.121:3300/0 conn(0x7f74d80c2860 0x7f74d80c2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 -- 172.21.15.90:0/3222598664 >> 172.21.15.90:0/3222598664 conn(0x7f74d806c0d0 msgr2=0x7f74d80c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:06.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 -- 172.21.15.90:0/3222598664 shutdown_connections 2023-03-03T06:18:06.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.571+0000 7f74df48f700 1 -- 172.21.15.90:0/3222598664 wait complete. 2023-03-03T06:18:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.572+0000 7f74df48f700 1 Processor -- start 2023-03-03T06:18:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.572+0000 7f74df48f700 1 -- start start 2023-03-03T06:18:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74df48f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d80bd220 0x7f74d80bae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74df48f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d80c2860 0x7f74d80bf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74df48f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d814fe60 0x7f74d81e0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74df48f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d80bb380 con 0x7f74d80c2860 2023-03-03T06:18:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74df48f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d81e0770 con 0x7f74d814fe60 2023-03-03T06:18:06.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74df48f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d80bb9c0 con 0x7f74d80bd220 2023-03-03T06:18:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d80c2860 0x7f74d80bf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d80c2860 0x7f74d80bf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33954/0 (socket says 172.21.15.90:33954) 2023-03-03T06:18:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 -- 172.21.15.90:0/3853836236 learned_addr learned my addr 172.21.15.90:0/3853836236 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dd22b700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d80bd220 0x7f74d80bae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dda2c700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d814fe60 0x7f74d81e0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 -- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d80bd220 msgr2=0x7f74d80bae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d80bd220 0x7f74d80bae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 -- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d814fe60 msgr2=0x7f74d81e0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d814fe60 0x7f74d81e0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 -- 172.21.15.90:0/3853836236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d000bd40 con 0x7f74d80c2860 2023-03-03T06:18:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74dca2a700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d80c2860 0x7f74d80bf040 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74c800b500 tx=0x7f74c800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74cdffb700 1 -- 172.21.15.90:0/3853836236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8011830 con 0x7f74d80c2860 2023-03-03T06:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74cdffb700 1 -- 172.21.15.90:0/3853836236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c8011990 con 0x7f74d80c2860 2023-03-03T06:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d80bdd30 con 0x7f74d80c2860 2023-03-03T06:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.573+0000 7f74cdffb700 1 -- 172.21.15.90:0/3853836236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8011d40 con 0x7f74d80c2860 2023-03-03T06:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.574+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d80bda50 con 0x7f74d80c2860 2023-03-03T06:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.574+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d80723b0 con 0x7f74d80c2860 2023-03-03T06:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.575+0000 7f74cdffb700 1 -- 172.21.15.90:0/3853836236 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f74c800f050 con 0x7f74d80c2860 2023-03-03T06:18:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.575+0000 7f74cdffb700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74bc0682f0 0x7f74bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.575+0000 7f74cdffb700 1 -- 172.21.15.90:0/3853836236 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f74c8085a30 con 0x7f74d80c2860 2023-03-03T06:18:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.575+0000 7f74dd22b700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74bc0682f0 0x7f74bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.576+0000 7f74dd22b700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74bc0682f0 0x7f74bc06a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006fd0 tx=0x7f74d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:06.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.579+0000 7f74cdffb700 1 -- 172.21.15.90:0/3853836236 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f74c80521a0 con 0x7f74d80c2860 2023-03-03T06:18:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:06.808+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f74d80bf580 con 0x7f74d80c2860 2023-03-03T06:18:07.763 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.762+0000 7f74cdffb700 1 -- 172.21.15.90:0/3853836236 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f74c80557c0 con 0x7f74d80c2860 2023-03-03T06:18:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.764+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74bc0682f0 msgr2=0x7f74bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.764+0000 7f74df48f700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74bc0682f0 0x7f74bc06a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006fd0 tx=0x7f74d4008040 comp rx=0 tx=0).stop 2023-03-03T06:18:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.764+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d80c2860 msgr2=0x7f74d80bf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.764+0000 7f74df48f700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d80c2860 0x7f74d80bf040 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74c800b500 tx=0x7f74c800b8c0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.764+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 shutdown_connections 2023-03-03T06:18:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.764+0000 7f74df48f700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d814fe60 0x7f74d81e0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.764+0000 7f74df48f700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d80bd220 0x7f74d80bae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.765+0000 7f74df48f700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f74bc0682f0 0x7f74bc06a7a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.765+0000 7f74df48f700 1 --2- 172.21.15.90:0/3853836236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74d80c2860 0x7f74d80bf040 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.765+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 >> 172.21.15.90:0/3853836236 conn(0x7f74d806c0d0 msgr2=0x7f74d806d990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.765+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 shutdown_connections 2023-03-03T06:18:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.765+0000 7f74df48f700 1 -- 172.21.15.90:0/3853836236 wait complete. 2023-03-03T06:18:07.781 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-03-03T06:18:07.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-03-03T06:18:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.908+0000 7f1e509be700 1 Processor -- start 2023-03-03T06:18:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.908+0000 7f1e509be700 1 -- start start 2023-03-03T06:18:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e509be700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1e4c0c2860 0x7f1e4c0c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e509be700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1e4c14fe60 0x7f1e4c156770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e509be700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e4c146310 con 0x7f1e4c0bd220 2023-03-03T06:18:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e509be700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4c06d420 con 0x7f1e4c0c2860 2023-03-03T06:18:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e509be700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4c14cc50 con 0x7f1e4c14fe60 2023-03-03T06:18:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4a59c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1e4c0c2860 0x7f1e4c0c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4a59c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1e4c0c2860 0x7f1e4c0c2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33960/0 (socket says 172.21.15.90:33960) 2023-03-03T06:18:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4a59c700 1 -- 172.21.15.90:0/2702769734 learned_addr learned my addr 172.21.15.90:0/2702769734 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4ad9d700 1 --2- 172.21.15.90:0/2702769734 >> v2:172.21.15.121:3300/0 conn(0x7f1e4c14fe60 0x7f1e4c156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4a59c700 1 -- 172.21.15.90:0/2702769734 >> v2:172.21.15.121:3300/0 conn(0x7f1e4c14fe60 msgr2=0x7f1e4c156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4a59c700 1 --2- 172.21.15.90:0/2702769734 >> v2:172.21.15.121:3300/0 conn(0x7f1e4c14fe60 0x7f1e4c156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4a59c700 1 -- 172.21.15.90:0/2702769734 >> v1:172.21.15.90:6789/0 conn(0x7f1e4c0bd220 legacy=0x7f1e4c14f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4a59c700 1 -- 172.21.15.90:0/2702769734 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c0bd770 con 0x7f1e4c0c2860 2023-03-03T06:18:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.909+0000 7f1e4ad9d700 1 --2- 172.21.15.90:0/2702769734 >> v2:172.21.15.121:3300/0 conn(0x7f1e4c14fe60 0x7f1e4c156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e4a59c700 1 --2- 172.21.15.90:0/2702769734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0c2860 0x7f1e4c0c2c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0beb80 tx=0x7f1e40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b3c0786d7404aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e4959a700 1 -- 172.21.15.90:0/2702769734 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4000b870 con 0x7f1e4c0c2860 2023-03-03T06:18:07.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e4959a700 1 -- 172.21.15.90:0/2702769734 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4000b9d0 con 0x7f1e4c0c2860 2023-03-03T06:18:07.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e4959a700 1 -- 172.21.15.90:0/2702769734 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4000bd60 con 0x7f1e4c0c2860 2023-03-03T06:18:07.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 -- 172.21.15.90:0/2702769734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0c2860 msgr2=0x7f1e4c0c2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:07.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 --2- 172.21.15.90:0/2702769734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0c2860 0x7f1e4c0c2c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0beb80 tx=0x7f1e40009bd0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 -- 172.21.15.90:0/2702769734 shutdown_connections 2023-03-03T06:18:07.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 --2- 172.21.15.90:0/2702769734 >> v2:172.21.15.121:3300/0 conn(0x7f1e4c14fe60 0x7f1e4c156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 --2- 172.21.15.90:0/2702769734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c0c2860 0x7f1e4c0c2c30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 -- 172.21.15.90:0/2702769734 >> 172.21.15.90:0/2702769734 conn(0x7f1e4c06c0d0 msgr2=0x7f1e4c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:07.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 -- 172.21.15.90:0/2702769734 shutdown_connections 2023-03-03T06:18:07.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.910+0000 7f1e509be700 1 -- 172.21.15.90:0/2702769734 wait complete. 2023-03-03T06:18:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.911+0000 7f1e509be700 1 Processor -- start 2023-03-03T06:18:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e509be700 1 -- start start 2023-03-03T06:18:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e509be700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e4c0bd220 0x7f1e4c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e509be700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e4c0c2860 0x7f1e4c144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e509be700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c14fe60 0x7f1e4c1e4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e509be700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c1e8430 con 0x7f1e4c14fe60 2023-03-03T06:18:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e509be700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c145430 con 0x7f1e4c0bd220 2023-03-03T06:18:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e509be700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c13d4b0 con 0x7f1e4c0c2860 2023-03-03T06:18:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e4ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c14fe60 0x7f1e4c1e4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e4ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c14fe60 0x7f1e4c1e4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33970/0 (socket says 172.21.15.90:33970) 2023-03-03T06:18:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e4ad9d700 1 -- 172.21.15.90:0/2190707461 learned_addr learned my addr 172.21.15.90:0/2190707461 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e4a59c700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e4c0bd220 0x7f1e4c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.912+0000 7f1e49d9b700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e4c0c2860 0x7f1e4c144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e4ad9d700 1 -- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e4c0c2860 msgr2=0x7f1e4c144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e4ad9d700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e4c0c2860 0x7f1e4c144ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e4ad9d700 1 -- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e4c0bd220 msgr2=0x7f1e4c14d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e4ad9d700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e4c0bd220 0x7f1e4c14d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e4ad9d700 1 -- 172.21.15.90:0/2190707461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e40009550 con 0x7f1e4c14fe60 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e4a59c700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e4c0bd220 0x7f1e4c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e4ad9d700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c14fe60 0x7f1e4c1e4ec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400cf20 tx=0x7f1e4400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e3affd700 1 -- 172.21.15.90:0/2190707461 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4400ae80 con 0x7f1e4c14fe60 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e3affd700 1 -- 172.21.15.90:0/2190707461 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e44015440 con 0x7f1e4c14fe60 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c0bdd30 con 0x7f1e4c14fe60 2023-03-03T06:18:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e3affd700 1 -- 172.21.15.90:0/2190707461 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e440157d0 con 0x7f1e4c14fe60 2023-03-03T06:18:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.913+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c0bda50 con 0x7f1e4c14fe60 2023-03-03T06:18:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.914+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4c0723b0 con 0x7f1e4c14fe60 2023-03-03T06:18:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.914+0000 7f1e3affd700 1 -- 172.21.15.90:0/2190707461 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1e4401c030 con 0x7f1e4c14fe60 2023-03-03T06:18:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.915+0000 7f1e3affd700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1e2c0682f0 0x7f1e2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.915+0000 7f1e4a59c700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1e2c0682f0 0x7f1e2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.915+0000 7f1e3affd700 1 -- 172.21.15.90:0/2190707461 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1e440861f0 con 0x7f1e4c14fe60 2023-03-03T06:18:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.915+0000 7f1e4a59c700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1e2c0682f0 0x7f1e2c06a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e40000c00 tx=0x7f1e40008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:07.919+0000 7f1e3affd700 1 -- 172.21.15.90:0/2190707461 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1e44052960 con 0x7f1e4c14fe60 2023-03-03T06:18:08.147 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.145+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1e4c04e750 con 0x7f1e4c14fe60 2023-03-03T06:18:08.770 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.768+0000 7f1e3affd700 1 -- 172.21.15.90:0/2190707461 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1e44055f80 con 0x7f1e4c14fe60 2023-03-03T06:18:08.772 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.770+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1e2c0682f0 msgr2=0x7f1e2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:08.772 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.770+0000 7f1e509be700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1e2c0682f0 0x7f1e2c06a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e40000c00 tx=0x7f1e40008030 comp rx=0 tx=0).stop 2023-03-03T06:18:08.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.770+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c14fe60 msgr2=0x7f1e4c1e4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:08.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.770+0000 7f1e509be700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c14fe60 0x7f1e4c1e4ec0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400cf20 tx=0x7f1e4400a340 comp rx=0 tx=0).stop 2023-03-03T06:18:08.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 shutdown_connections 2023-03-03T06:18:08.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1e2c0682f0 0x7f1e2c06a7a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e4c14fe60 0x7f1e4c1e4ec0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e4c0bd220 0x7f1e4c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.773 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 --2- 172.21.15.90:0/2190707461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e4c0c2860 0x7f1e4c144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 >> 172.21.15.90:0/2190707461 conn(0x7f1e4c06c0d0 msgr2=0x7f1e4c0bf460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:08.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 shutdown_connections 2023-03-03T06:18:08.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.771+0000 7f1e509be700 1 -- 172.21.15.90:0/2190707461 wait complete. 2023-03-03T06:18:08.785 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-03-03T06:18:08.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:18:08.906 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.905+0000 7f3f22a30700 1 Processor -- start 2023-03-03T06:18:08.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.905+0000 7f3f22a30700 1 -- start start 2023-03-03T06:18:08.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.905+0000 7f3f22a30700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3f1c06ea90 0x7f3f1c14f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.905+0000 7f3f22a30700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 0x7f3f1c159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.905+0000 7f3f22a30700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c153340 con 0x7f3f1c149420 2023-03-03T06:18:08.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.905+0000 7f3f22a30700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1c1497f0 con 0x7f3f1c06ea90 2023-03-03T06:18:08.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.905+0000 7f3f22a30700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c152750 con 0x7f3f1c14f600 2023-03-03T06:18:08.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1bfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3f1c149420 0x7f3f1c0c2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53180/0 (socket says 172.21.15.90:53180) 2023-03-03T06:18:08.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1bfff700 1 -- 172.21.15.90:0/3859306412 learned_addr learned my addr 172.21.15.90:0/3859306412 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:08.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289954554 0 0) 0x7f3f1c153340 con 0x7f3f1c149420 2023-03-03T06:18:08.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f040035b0 con 0x7f3f1c149420 2023-03-03T06:18:08.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/3859306412 >> v2:172.21.15.121:3300/0 conn(0x7f3f1c06ea90 0x7f3f1c14f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f20fcd700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 0x7f3f1c159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296380650 0 0) 0x7f3f040035b0 con 0x7f3f1c149420 2023-03-03T06:18:08.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 msgr2=0x7f3f1c159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:08.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 0x7f3f1c159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 >> v2:172.21.15.121:3300/0 conn(0x7f3f1c06ea90 msgr2=0x7f3f1c14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:08.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 --2- 172.21.15.90:0/3859306412 >> v2:172.21.15.121:3300/0 conn(0x7f3f1c06ea90 0x7f3f1c14f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c152a10 con 0x7f3f1c149420 2023-03-03T06:18:08.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f3f0c001670 con 0x7f3f1c149420 2023-03-03T06:18:08.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 >> v1:172.21.15.90:6789/0 conn(0x7f3f1c149420 legacy=0x7f3f1c0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:08.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 0x7f3f04003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f04004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f04004d90 0x7f3f04005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c1497f0 con 0x7f3f040040a0 2023-03-03T06:18:08.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c152750 con 0x7f3f04004d90 2023-03-03T06:18:08.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0400bca0 con 0x7f3f1c14f600 2023-03-03T06:18:08.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/3859306412 >> v2:172.21.15.121:3300/0 conn(0x7f3f1c06ea90 0x7f3f1c14f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:08.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f04004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1bfff700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 0x7f3f04003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f20fcd700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f04004d90 0x7f3f04005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 -- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 msgr2=0x7f3f04003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:08.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c14f600 0x7f3f04003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 -- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f04004d90 msgr2=0x7f3f04005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:08.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f04004d90 0x7f3f04005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.906+0000 7f3f1b7fe700 1 -- 172.21.15.90:0/3859306412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f040035b0 con 0x7f3f040040a0 2023-03-03T06:18:08.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f04004850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009930 tx=0x7f3f04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1260d17addfe23 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1000c070 con 0x7f3f040040a0 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f10004230 con 0x7f3f040040a0 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f1affd700 1 -- 172.21.15.90:0/3859306412 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100036a0 con 0x7f3f040040a0 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f22a30700 1 -- 172.21.15.90:0/3859306412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 msgr2=0x7f3f04004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f22a30700 1 --2- 172.21.15.90:0/3859306412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f04004850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009930 tx=0x7f3f04005880 comp rx=0 tx=0).stop 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f20fcd700 1 -- 172.21.15.90:0/3859306412 reap_dead start 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f22a30700 1 -- 172.21.15.90:0/3859306412 shutdown_connections 2023-03-03T06:18:08.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f22a30700 1 -- 172.21.15.90:0/3859306412 >> 172.21.15.90:0/3859306412 conn(0x7f3f1c06c0d0 msgr2=0x7f3f1c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:08.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f22a30700 1 -- 172.21.15.90:0/3859306412 shutdown_connections 2023-03-03T06:18:08.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.907+0000 7f3f22a30700 1 -- 172.21.15.90:0/3859306412 wait complete. 2023-03-03T06:18:08.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.909+0000 7f3f22a30700 1 Processor -- start 2023-03-03T06:18:08.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.909+0000 7f3f22a30700 1 -- start start 2023-03-03T06:18:08.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f1c0c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f1c14eb70 0x7f3f1c1d8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c1d8fc0 0x7f3f1c14e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c141530 con 0x7f3f040040a0 2023-03-03T06:18:08.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c145470 con 0x7f3f1c14eb70 2023-03-03T06:18:08.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c1462d0 con 0x7f3f1c1d8fc0 2023-03-03T06:18:08.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f1c0c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f1c0c0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33986/0 (socket says 172.21.15.90:33986) 2023-03-03T06:18:08.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 -- 172.21.15.90:0/1287159599 learned_addr learned my addr 172.21.15.90:0/1287159599 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:08.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f1c14eb70 0x7f3f1c1d8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f20fcd700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c1d8fc0 0x7f3f1c14e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 -- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c1d8fc0 msgr2=0x7f3f1c14e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:08.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c1d8fc0 0x7f3f1c14e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 -- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f1c14eb70 msgr2=0x7f3f1c1d8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:08.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f1c14eb70 0x7f3f1c1d8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 -- 172.21.15.90:0/1287159599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c152fd0 con 0x7f3f040040a0 2023-03-03T06:18:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f1bfff700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f1c0c0900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c007340 tx=0x7f3f0c007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f18ff9700 1 -- 172.21.15.90:0/1287159599 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00f640 con 0x7f3f040040a0 2023-03-03T06:18:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f18ff9700 1 -- 172.21.15.90:0/1287159599 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c00f7a0 con 0x7f3f040040a0 2023-03-03T06:18:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c152cf0 con 0x7f3f040040a0 2023-03-03T06:18:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f18ff9700 1 -- 172.21.15.90:0/1287159599 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00fb50 con 0x7f3f040040a0 2023-03-03T06:18:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.910+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c06e610 con 0x7f3f040040a0 2023-03-03T06:18:08.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.911+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c04e750 con 0x7f3f040040a0 2023-03-03T06:18:08.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.912+0000 7f3f18ff9700 1 -- 172.21.15.90:0/1287159599 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f0c00fcb0 con 0x7f3f040040a0 2023-03-03T06:18:08.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.912+0000 7f3f18ff9700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3f080682f0 0x7f3f0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:08.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.912+0000 7f3f18ff9700 1 -- 172.21.15.90:0/1287159599 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3f0c085ca0 con 0x7f3f040040a0 2023-03-03T06:18:08.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.912+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3f080682f0 0x7f3f0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:08.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.912+0000 7f3f1b7fe700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3f080682f0 0x7f3f0806a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000a010 tx=0x7f3f1000c000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:08.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:08.916+0000 7f3f18ff9700 1 -- 172.21.15.90:0/1287159599 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f0c08a070 con 0x7f3f040040a0 2023-03-03T06:18:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.140+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f1c14cae0 con 0x7f3f040040a0 2023-03-03T06:18:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.141+0000 7f3f18ff9700 1 -- 172.21.15.90:0/1287159599 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4648 (secure 0 0 0) 0x7f3f0c052670 con 0x7f3f040040a0 2023-03-03T06:18:09.143 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:09.143 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1770068404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1770068404},{"type":"v1","addr":"172.21.15.121:6834","nonce":1770068404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1411382210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":1411382210},{"type":"v1","addr":"172.21.15.90:6837","nonce":1411382210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4290,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2684520267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2684520267},{"type":"v1","addr":"172.21.15.121:6835","nonce":2684520267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T06:18:05.752643+0000","modified":"2023-03-03T06:18:08.761036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6836/3337533052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3337533052},{"type":"v1","addr":"172.21.15.90:6836","nonce":3337533052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:18:09.144 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3f080682f0 msgr2=0x7f3f0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3f080682f0 0x7f3f0806a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000a010 tx=0x7f3f1000c000 comp rx=0 tx=0).stop 2023-03-03T06:18:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 msgr2=0x7f3f1c0c0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f1c0c0900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c007340 tx=0x7f3f0c007650 comp rx=0 tx=0).stop 2023-03-03T06:18:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 shutdown_connections 2023-03-03T06:18:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3f1c1d8fc0 0x7f3f1c14e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3f080682f0 0x7f3f0806a7a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f040040a0 0x7f3f1c0c0900 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 --2- 172.21.15.90:0/1287159599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f1c14eb70 0x7f3f1c1d8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 >> 172.21.15.90:0/1287159599 conn(0x7f3f1c06c0d0 msgr2=0x7f3f1c06a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 shutdown_connections 2023-03-03T06:18:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.143+0000 7f3f22a30700 1 -- 172.21.15.90:0/1287159599 wait complete. 2023-03-03T06:18:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2023-03-03T06:18:09.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:09.276 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.274+0000 7f401f8c7700 1 Processor -- start 2023-03-03T06:18:09.276 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.274+0000 7f401f8c7700 1 -- start start 2023-03-03T06:18:09.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401f8c7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4018149420 0x7f40180c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401f8c7700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f40180c2da0 0x7f401814f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401f8c7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4018153340 con 0x7f401806ea90 2023-03-03T06:18:09.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401f8c7700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40181497f0 con 0x7f40180c2da0 2023-03-03T06:18:09.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401f8c7700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4018152750 con 0x7f4018149420 2023-03-03T06:18:09.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f40180c2da0 0x7f401814f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f40180c2da0 0x7f401814f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33996/0 (socket says 172.21.15.90:33996) 2023-03-03T06:18:09.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 -- 172.21.15.90:0/1081298880 learned_addr learned my addr 172.21.15.90:0/1081298880 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:09.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401de64700 1 --1- 172.21.15.90:0/1081298880 >> v1:172.21.15.90:6789/0 conn(0x7f401806ea90 0x7f401806a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53204/0 (socket says 172.21.15.90:53204) 2023-03-03T06:18:09.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401d663700 1 --2- 172.21.15.90:0/1081298880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4018149420 0x7f40180c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 -- 172.21.15.90:0/1081298880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4018149420 msgr2=0x7f40180c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:09.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 --2- 172.21.15.90:0/1081298880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4018149420 0x7f40180c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.281 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 -- 172.21.15.90:0/1081298880 >> v1:172.21.15.90:6789/0 conn(0x7f401806ea90 legacy=0x7f401806a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.281 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 -- 172.21.15.90:0/1081298880 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4018152a10 con 0x7f40180c2da0 2023-03-03T06:18:09.281 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401d663700 1 --2- 172.21.15.90:0/1081298880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4018149420 0x7f40180c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:09.281 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.275+0000 7f401ce62700 1 --2- 172.21.15.90:0/1081298880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f401814f0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f40080098f0 tx=0x7f4008009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64c8aa7bd178948 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.281 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f400ffff700 1 -- 172.21.15.90:0/1081298880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400800b8a0 con 0x7f40180c2da0 2023-03-03T06:18:09.281 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f400ffff700 1 -- 172.21.15.90:0/1081298880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400800ba00 con 0x7f40180c2da0 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f400ffff700 1 -- 172.21.15.90:0/1081298880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400800bd90 con 0x7f40180c2da0 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 -- 172.21.15.90:0/1081298880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 msgr2=0x7f401814f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 --2- 172.21.15.90:0/1081298880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f401814f0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f40080098f0 tx=0x7f4008009c00 comp rx=0 tx=0).stop 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 -- 172.21.15.90:0/1081298880 shutdown_connections 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 --2- 172.21.15.90:0/1081298880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4018149420 0x7f40180c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 --2- 172.21.15.90:0/1081298880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f401814f0c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 -- 172.21.15.90:0/1081298880 >> 172.21.15.90:0/1081298880 conn(0x7f401806c0d0 msgr2=0x7f40180c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:09.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 -- 172.21.15.90:0/1081298880 shutdown_connections 2023-03-03T06:18:09.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.276+0000 7f401f8c7700 1 -- 172.21.15.90:0/1081298880 wait complete. 2023-03-03T06:18:09.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.277+0000 7f401f8c7700 1 Processor -- start 2023-03-03T06:18:09.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.277+0000 7f401f8c7700 1 -- start start 2023-03-03T06:18:09.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401f8c7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401806ea90 0x7f40180a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401f8c7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f40180a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401f8c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4018149420 0x7f401806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401f8c7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40180a72f0 con 0x7f40180c2da0 2023-03-03T06:18:09.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401f8c7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401813fa60 con 0x7f4018149420 2023-03-03T06:18:09.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401f8c7700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40181e55c0 con 0x7f401806ea90 2023-03-03T06:18:09.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f40180a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f40180a6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34002/0 (socket says 172.21.15.90:34002) 2023-03-03T06:18:09.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 -- 172.21.15.90:0/3396686571 learned_addr learned my addr 172.21.15.90:0/3396686571 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:09.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401d663700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401806ea90 0x7f40180a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401de64700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4018149420 0x7f401806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 -- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401806ea90 msgr2=0x7f40180a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401806ea90 0x7f40180a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 -- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4018149420 msgr2=0x7f401806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4018149420 0x7f401806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 -- 172.21.15.90:0/3396686571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008009550 con 0x7f40180c2da0 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401de64700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4018149420 0x7f401806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.278+0000 7f401ce62700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f40180a6db0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4008009c40 tx=0x7f4008004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.279+0000 7f400dffb700 1 -- 172.21.15.90:0/3396686571 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40080036a0 con 0x7f40180c2da0 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.279+0000 7f400dffb700 1 -- 172.21.15.90:0/3396686571 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400801e070 con 0x7f40180c2da0 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.279+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4018152fd0 con 0x7f40180c2da0 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.279+0000 7f400dffb700 1 -- 172.21.15.90:0/3396686571 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008003a60 con 0x7f40180c2da0 2023-03-03T06:18:09.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.279+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4018152cf0 con 0x7f40180c2da0 2023-03-03T06:18:09.289 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.280+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401806e610 con 0x7f40180c2da0 2023-03-03T06:18:09.289 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.280+0000 7f400dffb700 1 -- 172.21.15.90:0/3396686571 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f400800f050 con 0x7f40180c2da0 2023-03-03T06:18:09.289 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.280+0000 7f400dffb700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3ffc0682f0 0x7f3ffc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.289 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.281+0000 7f400dffb700 1 -- 172.21.15.90:0/3396686571 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4008020070 con 0x7f40180c2da0 2023-03-03T06:18:09.289 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.281+0000 7f401d663700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3ffc0682f0 0x7f3ffc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.289 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.281+0000 7f401d663700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3ffc0682f0 0x7f3ffc06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4014009990 tx=0x7f4014008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.290 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.285+0000 7f400dffb700 1 -- 172.21.15.90:0/3396686571 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4008053a10 con 0x7f40180c2da0 2023-03-03T06:18:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.468+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f401806aae0 con 0x7f40180c2da0 2023-03-03T06:18:09.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.469+0000 7f400dffb700 1 -- 172.21.15.90:0/3396686571 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17811 (secure 0 0 0) 0x7f4008057030 con 0x7f40180c2da0 2023-03-03T06:18:09.471 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:09.472 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":23,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:06.754461+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T06:18:03.400568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T06:18:04.718088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:09.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.471+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3ffc0682f0 msgr2=0x7f3ffc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.471+0000 7f401f8c7700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3ffc0682f0 0x7f3ffc06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4014009990 tx=0x7f4014008040 comp rx=0 tx=0).stop 2023-03-03T06:18:09.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.471+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 msgr2=0x7f40180a6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.471+0000 7f401f8c7700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f40180a6db0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4008009c40 tx=0x7f4008004750 comp rx=0 tx=0).stop 2023-03-03T06:18:09.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 shutdown_connections 2023-03-03T06:18:09.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4018149420 0x7f401806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401806ea90 0x7f40180a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3ffc0682f0 0x7f3ffc06a7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 --2- 172.21.15.90:0/3396686571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40180c2da0 0x7f40180a6db0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 >> 172.21.15.90:0/3396686571 conn(0x7f401806c0d0 msgr2=0x7f40180b7070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:09.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 shutdown_connections 2023-03-03T06:18:09.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.472+0000 7f401f8c7700 1 -- 172.21.15.90:0/3396686571 wait complete. 2023-03-03T06:18:09.485 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T06:18:09.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T06:18:09.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.603+0000 7fae665c2700 1 Processor -- start 2023-03-03T06:18:09.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.603+0000 7fae665c2700 1 -- start start 2023-03-03T06:18:09.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae665c2700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fae6014fe60 0x7fae600c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae665c2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae60146310 con 0x7fae600bd220 2023-03-03T06:18:09.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae665c2700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6006d420 con 0x7fae600c2860 2023-03-03T06:18:09.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae665c2700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae6014cc50 con 0x7fae6014fe60 2023-03-03T06:18:09.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae5f7fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fae6014fe60 0x7fae600c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae5f7fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fae6014fe60 0x7fae600c2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34010/0 (socket says 172.21.15.90:34010) 2023-03-03T06:18:09.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae5f7fe700 1 -- 172.21.15.90:0/1847403820 learned_addr learned my addr 172.21.15.90:0/1847403820 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:09.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae5f7fe700 1 -- 172.21.15.90:0/1847403820 >> v1:172.21.15.121:6789/0 conn(0x7fae600c2860 legacy=0x7fae6014f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:09.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae5f7fe700 1 -- 172.21.15.90:0/1847403820 >> v1:172.21.15.90:6789/0 conn(0x7fae600bd220 legacy=0x7fae60156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.604+0000 7fae5f7fe700 1 -- 172.21.15.90:0/1847403820 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae600bd770 con 0x7fae6014fe60 2023-03-03T06:18:09.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae5f7fe700 1 --2- 172.21.15.90:0/1847403820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae600c2c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fae5000bb90 tx=0x7fae5000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e494eb26c764693b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae5effd700 1 -- 172.21.15.90:0/1847403820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5000dd50 con 0x7fae6014fe60 2023-03-03T06:18:09.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae5effd700 1 -- 172.21.15.90:0/1847403820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50004030 con 0x7fae6014fe60 2023-03-03T06:18:09.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae5effd700 1 -- 172.21.15.90:0/1847403820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae500043c0 con 0x7fae6014fe60 2023-03-03T06:18:09.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae665c2700 1 -- 172.21.15.90:0/1847403820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 msgr2=0x7fae600c2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae665c2700 1 --2- 172.21.15.90:0/1847403820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae600c2c30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fae5000bb90 tx=0x7fae5000bea0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae665c2700 1 -- 172.21.15.90:0/1847403820 shutdown_connections 2023-03-03T06:18:09.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae665c2700 1 --2- 172.21.15.90:0/1847403820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae600c2c30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae665c2700 1 -- 172.21.15.90:0/1847403820 >> 172.21.15.90:0/1847403820 conn(0x7fae6006c0d0 msgr2=0x7fae600c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:09.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae665c2700 1 -- 172.21.15.90:0/1847403820 shutdown_connections 2023-03-03T06:18:09.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.605+0000 7fae665c2700 1 -- 172.21.15.90:0/1847403820 wait complete. 2023-03-03T06:18:09.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.606+0000 7fae665c2700 1 Processor -- start 2023-03-03T06:18:09.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.606+0000 7fae665c2700 1 -- start start 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae665c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae600bd220 0x7fae60149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae665c2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae600c2860 0x7fae60141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae665c2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae6014b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae665c2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600bbe10 con 0x7fae6014fe60 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae665c2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae60146c60 con 0x7fae600bd220 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae665c2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae600bb770 con 0x7fae600c2860 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae6014b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae6014b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34018/0 (socket says 172.21.15.90:34018) 2023-03-03T06:18:09.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 -- 172.21.15.90:0/230525956 learned_addr learned my addr 172.21.15.90:0/230525956 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:09.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 -- 172.21.15.90:0/230525956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae600c2860 msgr2=0x7fae60141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae600c2860 0x7fae60141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 -- 172.21.15.90:0/230525956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae600bd220 msgr2=0x7fae60149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae600bd220 0x7fae60149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.607+0000 7fae64b5f700 1 -- 172.21.15.90:0/230525956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5000b7f0 con 0x7fae6014fe60 2023-03-03T06:18:09.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.608+0000 7fae64b5f700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae6014b360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae5800cb50 tx=0x7fae5800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.608+0000 7fae5cff9700 1 -- 172.21.15.90:0/230525956 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58003c00 con 0x7fae6014fe60 2023-03-03T06:18:09.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.608+0000 7fae5cff9700 1 -- 172.21.15.90:0/230525956 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae58003d60 con 0x7fae6014fe60 2023-03-03T06:18:09.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.608+0000 7fae5cff9700 1 -- 172.21.15.90:0/230525956 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5800ab70 con 0x7fae6014fe60 2023-03-03T06:18:09.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.608+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae600bdd30 con 0x7fae6014fe60 2023-03-03T06:18:09.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.608+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae600bda50 con 0x7fae6014fe60 2023-03-03T06:18:09.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.609+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae600723b0 con 0x7fae6014fe60 2023-03-03T06:18:09.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.609+0000 7fae5cff9700 1 -- 172.21.15.90:0/230525956 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae58010050 con 0x7fae6014fe60 2023-03-03T06:18:09.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.609+0000 7fae5cff9700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fae4c0682f0 0x7fae4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.609+0000 7fae5cff9700 1 -- 172.21.15.90:0/230525956 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fae58085a50 con 0x7fae6014fe60 2023-03-03T06:18:09.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.609+0000 7fae5ffff700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fae4c0682f0 0x7fae4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.610+0000 7fae5ffff700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fae4c0682f0 0x7fae4c06a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fae54003df0 tx=0x7fae54007a40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.613+0000 7fae5cff9700 1 -- 172.21.15.90:0/230525956 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae580522b0 con 0x7fae6014fe60 2023-03-03T06:18:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.793+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 --> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae601462e0 con 0x7fae4c0682f0 2023-03-03T06:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.801+0000 7fae5cff9700 1 -- 172.21.15.90:0/230525956 <== mgr.4108 v2:172.21.15.90:6832/96475 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286674 (secure 0 0 0) 0x7fae601462e0 con 0x7fae4c0682f0 2023-03-03T06:18:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:09.808 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fae4c0682f0 msgr2=0x7fae4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.808 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fae4c0682f0 0x7fae4c06a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fae54003df0 tx=0x7fae54007a40 comp rx=0 tx=0).stop 2023-03-03T06:18:09.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 msgr2=0x7fae6014b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae6014b360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae5800cb50 tx=0x7fae5800cf10 comp rx=0 tx=0).stop 2023-03-03T06:18:09.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 shutdown_connections 2023-03-03T06:18:09.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fae4c0682f0 0x7fae4c06a7a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae6014fe60 0x7fae6014b360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae600bd220 0x7fae60149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 --2- 172.21.15.90:0/230525956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae600c2860 0x7fae60141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 >> 172.21.15.90:0/230525956 conn(0x7fae6006c0d0 msgr2=0x7fae600b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:09.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 shutdown_connections 2023-03-03T06:18:09.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.805+0000 7fae665c2700 1 -- 172.21.15.90:0/230525956 wait complete. 2023-03-03T06:18:09.810 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-03-03T06:18:09.819 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-03-03T06:18:08.074563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66120,"kb_used_data":3016,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":754908600,"statfs":{"total":773094113280,"available":773026406400,"internally_reserved":0,"allocated":3088384,"data_stored":1848964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8],"upper_bound":8},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000751"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773776+0000","last_change":"2023-03-03T06:18:05.768223+0000","last_active":"2023-03-03T06:18:05.773776+0000","last_peered":"2023-03-03T06:18:05.773776+0000","last_clean":"2023-03-03T06:18:05.773776+0000","last_became_active":"2023-03-03T06:18:05.767943+0000","last_became_peered":"2023-03-03T06:18:05.767943+0000","last_unstale":"2023-03-03T06:18:05.773776+0000","last_undegraded":"2023-03-03T06:18:05.773776+0000","last_fullsized":"2023-03-03T06:18:05.773776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:38.546721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767372+0000","last_change":"2023-03-03T06:18:05.779938+0000","last_active":"2023-03-03T06:18:06.767372+0000","last_peered":"2023-03-03T06:18:06.767372+0000","last_clean":"2023-03-03T06:18:06.767372+0000","last_became_active":"2023-03-03T06:18:05.779669+0000","last_became_peered":"2023-03-03T06:18:05.779669+0000","last_unstale":"2023-03-03T06:18:06.767372+0000","last_undegraded":"2023-03-03T06:18:06.767372+0000","last_fullsized":"2023-03-03T06:18:06.767372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780808+0000","last_change":"2023-03-03T06:18:05.773328+0000","last_active":"2023-03-03T06:18:05.780808+0000","last_peered":"2023-03-03T06:18:05.780808+0000","last_clean":"2023-03-03T06:18:05.780808+0000","last_became_active":"2023-03-03T06:18:05.773065+0000","last_became_peered":"2023-03-03T06:18:05.773065+0000","last_unstale":"2023-03-03T06:18:05.780808+0000","last_undegraded":"2023-03-03T06:18:05.780808+0000","last_fullsized":"2023-03-03T06:18:05.780808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:58.062737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.778686+0000","last_change":"2023-03-03T06:18:05.773425+0000","last_active":"2023-03-03T06:18:05.778686+0000","last_peered":"2023-03-03T06:18:05.778686+0000","last_clean":"2023-03-03T06:18:05.778686+0000","last_became_active":"2023-03-03T06:18:05.772997+0000","last_became_peered":"2023-03-03T06:18:05.772997+0000","last_unstale":"2023-03-03T06:18:05.778686+0000","last_undegraded":"2023-03-03T06:18:05.778686+0000","last_fullsized":"2023-03-03T06:18:05.778686+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:43.144401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773462+0000","last_change":"2023-03-03T06:18:05.771575+0000","last_active":"2023-03-03T06:18:05.773462+0000","last_peered":"2023-03-03T06:18:05.773462+0000","last_clean":"2023-03-03T06:18:05.773462+0000","last_became_active":"2023-03-03T06:18:05.771326+0000","last_became_peered":"2023-03-03T06:18:05.771326+0000","last_unstale":"2023-03-03T06:18:05.773462+0000","last_undegraded":"2023-03-03T06:18:05.773462+0000","last_fullsized":"2023-03-03T06:18:05.773462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781831+0000","last_change":"2023-03-03T06:18:05.773873+0000","last_active":"2023-03-03T06:18:05.781831+0000","last_peered":"2023-03-03T06:18:05.781831+0000","last_clean":"2023-03-03T06:18:05.781831+0000","last_became_active":"2023-03-03T06:18:05.773610+0000","last_became_peered":"2023-03-03T06:18:05.773610+0000","last_unstale":"2023-03-03T06:18:05.781831+0000","last_undegraded":"2023-03-03T06:18:05.781831+0000","last_fullsized":"2023-03-03T06:18:05.781831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:24:21.809616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766274+0000","last_change":"2023-03-03T06:18:05.779587+0000","last_active":"2023-03-03T06:18:06.766274+0000","last_peered":"2023-03-03T06:18:06.766274+0000","last_clean":"2023-03-03T06:18:06.766274+0000","last_became_active":"2023-03-03T06:18:05.779147+0000","last_became_peered":"2023-03-03T06:18:05.779147+0000","last_unstale":"2023-03-03T06:18:06.766274+0000","last_undegraded":"2023-03-03T06:18:06.766274+0000","last_fullsized":"2023-03-03T06:18:06.766274+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766529+0000","last_change":"2023-03-03T06:18:05.774633+0000","last_active":"2023-03-03T06:18:06.766529+0000","last_peered":"2023-03-03T06:18:06.766529+0000","last_clean":"2023-03-03T06:18:06.766529+0000","last_became_active":"2023-03-03T06:18:05.773725+0000","last_became_peered":"2023-03-03T06:18:05.773725+0000","last_unstale":"2023-03-03T06:18:06.766529+0000","last_undegraded":"2023-03-03T06:18:06.766529+0000","last_fullsized":"2023-03-03T06:18:06.766529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.772498+0000","last_change":"2023-03-03T06:18:05.770503+0000","last_active":"2023-03-03T06:18:05.772498+0000","last_peered":"2023-03-03T06:18:05.772498+0000","last_clean":"2023-03-03T06:18:05.772498+0000","last_became_active":"2023-03-03T06:18:05.770093+0000","last_became_peered":"2023-03-03T06:18:05.770093+0000","last_unstale":"2023-03-03T06:18:05.772498+0000","last_undegraded":"2023-03-03T06:18:05.772498+0000","last_fullsized":"2023-03-03T06:18:05.772498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767476+0000","last_change":"2023-03-03T06:18:05.774425+0000","last_active":"2023-03-03T06:18:06.767476+0000","last_peered":"2023-03-03T06:18:06.767476+0000","last_clean":"2023-03-03T06:18:06.767476+0000","last_became_active":"2023-03-03T06:18:05.774137+0000","last_became_peered":"2023-03-03T06:18:05.774137+0000","last_unstale":"2023-03-03T06:18:06.767476+0000","last_undegraded":"2023-03-03T06:18:06.767476+0000","last_fullsized":"2023-03-03T06:18:06.767476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783745+0000","last_change":"2023-03-03T06:18:05.779203+0000","last_active":"2023-03-03T06:18:05.783745+0000","last_peered":"2023-03-03T06:18:05.783745+0000","last_clean":"2023-03-03T06:18:05.783745+0000","last_became_active":"2023-03-03T06:18:05.778887+0000","last_became_peered":"2023-03-03T06:18:05.778887+0000","last_unstale":"2023-03-03T06:18:05.783745+0000","last_undegraded":"2023-03-03T06:18:05.783745+0000","last_fullsized":"2023-03-03T06:18:05.783745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.774328+0000","last_change":"2023-03-03T06:18:05.771290+0000","last_active":"2023-03-03T06:18:05.774328+0000","last_peered":"2023-03-03T06:18:05.774328+0000","last_clean":"2023-03-03T06:18:05.774328+0000","last_became_active":"2023-03-03T06:18:05.771073+0000","last_became_peered":"2023-03-03T06:18:05.771073+0000","last_unstale":"2023-03-03T06:18:05.774328+0000","last_undegraded":"2023-03-03T06:18:05.774328+0000","last_fullsized":"2023-03-03T06:18:05.774328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:01.951025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.774365+0000","last_change":"2023-03-03T06:18:05.772434+0000","last_active":"2023-03-03T06:18:05.774365+0000","last_peered":"2023-03-03T06:18:05.774365+0000","last_clean":"2023-03-03T06:18:05.774365+0000","last_became_active":"2023-03-03T06:18:05.772171+0000","last_became_peered":"2023-03-03T06:18:05.772171+0000","last_unstale":"2023-03-03T06:18:05.774365+0000","last_undegraded":"2023-03-03T06:18:05.774365+0000","last_fullsized":"2023-03-03T06:18:05.774365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:43.200309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766717+0000","last_change":"2023-03-03T06:18:05.775751+0000","last_active":"2023-03-03T06:18:06.766717+0000","last_peered":"2023-03-03T06:18:06.766717+0000","last_clean":"2023-03-03T06:18:06.766717+0000","last_became_active":"2023-03-03T06:18:05.775268+0000","last_became_peered":"2023-03-03T06:18:05.775268+0000","last_unstale":"2023-03-03T06:18:06.766717+0000","last_undegraded":"2023-03-03T06:18:06.766717+0000","last_fullsized":"2023-03-03T06:18:06.766717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766405+0000","last_change":"2023-03-03T06:18:05.779415+0000","last_active":"2023-03-03T06:18:06.766405+0000","last_peered":"2023-03-03T06:18:06.766405+0000","last_clean":"2023-03-03T06:18:06.766405+0000","last_became_active":"2023-03-03T06:18:05.778767+0000","last_became_peered":"2023-03-03T06:18:05.778767+0000","last_unstale":"2023-03-03T06:18:06.766405+0000","last_undegraded":"2023-03-03T06:18:06.766405+0000","last_fullsized":"2023-03-03T06:18:06.766405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.777740+0000","last_change":"2023-03-03T06:18:05.768282+0000","last_active":"2023-03-03T06:18:05.777740+0000","last_peered":"2023-03-03T06:18:05.777740+0000","last_clean":"2023-03-03T06:18:05.777740+0000","last_became_active":"2023-03-03T06:18:05.767972+0000","last_became_peered":"2023-03-03T06:18:05.767972+0000","last_unstale":"2023-03-03T06:18:05.777740+0000","last_undegraded":"2023-03-03T06:18:05.777740+0000","last_fullsized":"2023-03-03T06:18:05.777740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:46.885266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.772479+0000","last_change":"2023-03-03T06:18:05.770349+0000","last_active":"2023-03-03T06:18:05.772479+0000","last_peered":"2023-03-03T06:18:05.772479+0000","last_clean":"2023-03-03T06:18:05.772479+0000","last_became_active":"2023-03-03T06:18:05.769801+0000","last_became_peered":"2023-03-03T06:18:05.769801+0000","last_unstale":"2023-03-03T06:18:05.772479+0000","last_undegraded":"2023-03-03T06:18:05.772479+0000","last_fullsized":"2023-03-03T06:18:05.772479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.777774+0000","last_change":"2023-03-03T06:18:05.769983+0000","last_active":"2023-03-03T06:18:05.777774+0000","last_peered":"2023-03-03T06:18:05.777774+0000","last_clean":"2023-03-03T06:18:05.777774+0000","last_became_active":"2023-03-03T06:18:05.769713+0000","last_became_peered":"2023-03-03T06:18:05.769713+0000","last_unstale":"2023-03-03T06:18:05.777774+0000","last_undegraded":"2023-03-03T06:18:05.777774+0000","last_fullsized":"2023-03-03T06:18:05.777774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:48.612674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.774338+0000","last_change":"2023-03-03T06:18:05.772674+0000","last_active":"2023-03-03T06:18:05.774338+0000","last_peered":"2023-03-03T06:18:05.774338+0000","last_clean":"2023-03-03T06:18:05.774338+0000","last_became_active":"2023-03-03T06:18:05.772420+0000","last_became_peered":"2023-03-03T06:18:05.772420+0000","last_unstale":"2023-03-03T06:18:05.774338+0000","last_undegraded":"2023-03-03T06:18:05.774338+0000","last_fullsized":"2023-03-03T06:18:05.774338+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:17.201682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783869+0000","last_change":"2023-03-03T06:18:05.779465+0000","last_active":"2023-03-03T06:18:05.783869+0000","last_peered":"2023-03-03T06:18:05.783869+0000","last_clean":"2023-03-03T06:18:05.783869+0000","last_became_active":"2023-03-03T06:18:05.779177+0000","last_became_peered":"2023-03-03T06:18:05.779177+0000","last_unstale":"2023-03-03T06:18:05.783869+0000","last_undegraded":"2023-03-03T06:18:05.783869+0000","last_fullsized":"2023-03-03T06:18:05.783869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766718+0000","last_change":"2023-03-03T06:18:05.769339+0000","last_active":"2023-03-03T06:18:06.766718+0000","last_peered":"2023-03-03T06:18:06.766718+0000","last_clean":"2023-03-03T06:18:06.766718+0000","last_became_active":"2023-03-03T06:18:05.768277+0000","last_became_peered":"2023-03-03T06:18:05.768277+0000","last_unstale":"2023-03-03T06:18:06.766718+0000","last_undegraded":"2023-03-03T06:18:06.766718+0000","last_fullsized":"2023-03-03T06:18:06.766718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781531+0000","last_change":"2023-03-03T06:18:05.780097+0000","last_active":"2023-03-03T06:18:05.781531+0000","last_peered":"2023-03-03T06:18:05.781531+0000","last_clean":"2023-03-03T06:18:05.781531+0000","last_became_active":"2023-03-03T06:18:05.779838+0000","last_became_peered":"2023-03-03T06:18:05.779838+0000","last_unstale":"2023-03-03T06:18:05.781531+0000","last_undegraded":"2023-03-03T06:18:05.781531+0000","last_fullsized":"2023-03-03T06:18:05.781531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761222+0000","last_change":"2023-03-03T06:18:05.772869+0000","last_active":"2023-03-03T06:18:06.761222+0000","last_peered":"2023-03-03T06:18:06.761222+0000","last_clean":"2023-03-03T06:18:06.761222+0000","last_became_active":"2023-03-03T06:18:05.772498+0000","last_became_peered":"2023-03-03T06:18:05.772498+0000","last_unstale":"2023-03-03T06:18:06.761222+0000","last_undegraded":"2023-03-03T06:18:06.761222+0000","last_fullsized":"2023-03-03T06:18:06.761222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766506+0000","last_change":"2023-03-03T06:18:05.769326+0000","last_active":"2023-03-03T06:18:06.766506+0000","last_peered":"2023-03-03T06:18:06.766506+0000","last_clean":"2023-03-03T06:18:06.766506+0000","last_became_active":"2023-03-03T06:18:05.768781+0000","last_became_peered":"2023-03-03T06:18:05.768781+0000","last_unstale":"2023-03-03T06:18:06.766506+0000","last_undegraded":"2023-03-03T06:18:06.766506+0000","last_fullsized":"2023-03-03T06:18:06.766506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782707+0000","last_change":"2023-03-03T06:18:05.777824+0000","last_active":"2023-03-03T06:18:05.782707+0000","last_peered":"2023-03-03T06:18:05.782707+0000","last_clean":"2023-03-03T06:18:05.782707+0000","last_became_active":"2023-03-03T06:18:05.777221+0000","last_became_peered":"2023-03-03T06:18:05.777221+0000","last_unstale":"2023-03-03T06:18:05.782707+0000","last_undegraded":"2023-03-03T06:18:05.782707+0000","last_fullsized":"2023-03-03T06:18:05.782707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782704+0000","last_change":"2023-03-03T06:18:05.777434+0000","last_active":"2023-03-03T06:18:05.782704+0000","last_peered":"2023-03-03T06:18:05.782704+0000","last_clean":"2023-03-03T06:18:05.782704+0000","last_became_active":"2023-03-03T06:18:05.776857+0000","last_became_peered":"2023-03-03T06:18:05.776857+0000","last_unstale":"2023-03-03T06:18:05.782704+0000","last_undegraded":"2023-03-03T06:18:05.782704+0000","last_fullsized":"2023-03-03T06:18:05.782704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767237+0000","last_change":"2023-03-03T06:18:05.776802+0000","last_active":"2023-03-03T06:18:06.767237+0000","last_peered":"2023-03-03T06:18:06.767237+0000","last_clean":"2023-03-03T06:18:06.767237+0000","last_became_active":"2023-03-03T06:18:05.776531+0000","last_became_peered":"2023-03-03T06:18:05.776531+0000","last_unstale":"2023-03-03T06:18:06.767237+0000","last_undegraded":"2023-03-03T06:18:06.767237+0000","last_fullsized":"2023-03-03T06:18:06.767237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781661+0000","last_change":"2023-03-03T06:18:05.374916+0000","last_active":"2023-03-03T06:18:05.781661+0000","last_peered":"2023-03-03T06:18:05.781661+0000","last_clean":"2023-03-03T06:18:05.781661+0000","last_became_active":"2023-03-03T06:18:05.374663+0000","last_became_peered":"2023-03-03T06:18:05.374663+0000","last_unstale":"2023-03-03T06:18:05.781661+0000","last_undegraded":"2023-03-03T06:18:05.781661+0000","last_fullsized":"2023-03-03T06:18:05.781661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:19:22.426709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761867+0000","last_change":"2023-03-03T06:18:05.778939+0000","last_active":"2023-03-03T06:18:06.761867+0000","last_peered":"2023-03-03T06:18:06.761867+0000","last_clean":"2023-03-03T06:18:06.761867+0000","last_became_active":"2023-03-03T06:18:05.778350+0000","last_became_peered":"2023-03-03T06:18:05.778350+0000","last_unstale":"2023-03-03T06:18:06.761867+0000","last_undegraded":"2023-03-03T06:18:06.761867+0000","last_fullsized":"2023-03-03T06:18:06.761867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.770605+0000","last_change":"2023-03-03T06:18:05.369187+0000","last_active":"2023-03-03T06:18:05.770605+0000","last_peered":"2023-03-03T06:18:05.770605+0000","last_clean":"2023-03-03T06:18:05.770605+0000","last_became_active":"2023-03-03T06:18:05.368706+0000","last_became_peered":"2023-03-03T06:18:05.368706+0000","last_unstale":"2023-03-03T06:18:05.770605+0000","last_undegraded":"2023-03-03T06:18:05.770605+0000","last_fullsized":"2023-03-03T06:18:05.770605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773787+0000","last_change":"2023-03-03T06:18:05.771810+0000","last_active":"2023-03-03T06:18:05.773787+0000","last_peered":"2023-03-03T06:18:05.773787+0000","last_clean":"2023-03-03T06:18:05.773787+0000","last_became_active":"2023-03-03T06:18:05.771484+0000","last_became_peered":"2023-03-03T06:18:05.771484+0000","last_unstale":"2023-03-03T06:18:05.773787+0000","last_undegraded":"2023-03-03T06:18:05.773787+0000","last_fullsized":"2023-03-03T06:18:05.773787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761169+0000","last_change":"2023-03-03T06:18:05.378838+0000","last_active":"2023-03-03T06:18:06.761169+0000","last_peered":"2023-03-03T06:18:06.761169+0000","last_clean":"2023-03-03T06:18:06.761169+0000","last_became_active":"2023-03-03T06:18:05.378525+0000","last_became_peered":"2023-03-03T06:18:05.378525+0000","last_unstale":"2023-03-03T06:18:06.761169+0000","last_undegraded":"2023-03-03T06:18:06.761169+0000","last_fullsized":"2023-03-03T06:18:06.761169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783276+0000","last_change":"2023-03-03T06:18:05.776315+0000","last_active":"2023-03-03T06:18:05.783276+0000","last_peered":"2023-03-03T06:18:05.783276+0000","last_clean":"2023-03-03T06:18:05.783276+0000","last_became_active":"2023-03-03T06:18:05.775728+0000","last_became_peered":"2023-03-03T06:18:05.775728+0000","last_unstale":"2023-03-03T06:18:05.783276+0000","last_undegraded":"2023-03-03T06:18:05.783276+0000","last_fullsized":"2023-03-03T06:18:05.783276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:17.287057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.763700+0000","last_change":"2023-03-03T06:18:05.378086+0000","last_active":"2023-03-03T06:18:05.763700+0000","last_peered":"2023-03-03T06:18:05.763700+0000","last_clean":"2023-03-03T06:18:05.763700+0000","last_became_active":"2023-03-03T06:18:05.377265+0000","last_became_peered":"2023-03-03T06:18:05.377265+0000","last_unstale":"2023-03-03T06:18:05.763700+0000","last_undegraded":"2023-03-03T06:18:05.763700+0000","last_fullsized":"2023-03-03T06:18:05.763700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761128+0000","last_change":"2023-03-03T06:18:05.778565+0000","last_active":"2023-03-03T06:18:06.761128+0000","last_peered":"2023-03-03T06:18:06.761128+0000","last_clean":"2023-03-03T06:18:06.761128+0000","last_became_active":"2023-03-03T06:18:05.778231+0000","last_became_peered":"2023-03-03T06:18:05.778231+0000","last_unstale":"2023-03-03T06:18:06.761128+0000","last_undegraded":"2023-03-03T06:18:06.761128+0000","last_fullsized":"2023-03-03T06:18:06.761128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767346+0000","last_change":"2023-03-03T06:18:05.370256+0000","last_active":"2023-03-03T06:18:06.767346+0000","last_peered":"2023-03-03T06:18:06.767346+0000","last_clean":"2023-03-03T06:18:06.767346+0000","last_became_active":"2023-03-03T06:18:05.369970+0000","last_became_peered":"2023-03-03T06:18:05.369970+0000","last_unstale":"2023-03-03T06:18:06.767346+0000","last_undegraded":"2023-03-03T06:18:06.767346+0000","last_fullsized":"2023-03-03T06:18:06.767346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761149+0000","last_change":"2023-03-03T06:18:05.766019+0000","last_active":"2023-03-03T06:18:06.761149+0000","last_peered":"2023-03-03T06:18:06.761149+0000","last_clean":"2023-03-03T06:18:06.761149+0000","last_became_active":"2023-03-03T06:18:05.765768+0000","last_became_peered":"2023-03-03T06:18:05.765768+0000","last_unstale":"2023-03-03T06:18:06.761149+0000","last_undegraded":"2023-03-03T06:18:06.761149+0000","last_fullsized":"2023-03-03T06:18:06.761149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766991+0000","last_change":"2023-03-03T06:18:05.367491+0000","last_active":"2023-03-03T06:18:06.766991+0000","last_peered":"2023-03-03T06:18:06.766991+0000","last_clean":"2023-03-03T06:18:06.766991+0000","last_became_active":"2023-03-03T06:18:05.367177+0000","last_became_peered":"2023-03-03T06:18:05.367177+0000","last_unstale":"2023-03-03T06:18:06.766991+0000","last_undegraded":"2023-03-03T06:18:06.766991+0000","last_fullsized":"2023-03-03T06:18:06.766991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:55.267658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782754+0000","last_change":"2023-03-03T06:18:05.778585+0000","last_active":"2023-03-03T06:18:05.782754+0000","last_peered":"2023-03-03T06:18:05.782754+0000","last_clean":"2023-03-03T06:18:05.782754+0000","last_became_active":"2023-03-03T06:18:05.777053+0000","last_became_peered":"2023-03-03T06:18:05.777053+0000","last_unstale":"2023-03-03T06:18:05.782754+0000","last_undegraded":"2023-03-03T06:18:05.782754+0000","last_fullsized":"2023-03-03T06:18:05.782754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.786362+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:05.786362+0000","last_peered":"2023-03-03T06:18:05.786362+0000","last_clean":"2023-03-03T06:18:05.786362+0000","last_became_active":"2023-03-03T06:18:05.372051+0000","last_became_peered":"2023-03-03T06:18:05.372051+0000","last_unstale":"2023-03-03T06:18:05.786362+0000","last_undegraded":"2023-03-03T06:18:05.786362+0000","last_fullsized":"2023-03-03T06:18:05.786362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783758+0000","last_change":"2023-03-03T06:18:05.779300+0000","last_active":"2023-03-03T06:18:05.783758+0000","last_peered":"2023-03-03T06:18:05.783758+0000","last_clean":"2023-03-03T06:18:05.783758+0000","last_became_active":"2023-03-03T06:18:05.778878+0000","last_became_peered":"2023-03-03T06:18:05.778878+0000","last_unstale":"2023-03-03T06:18:05.783758+0000","last_undegraded":"2023-03-03T06:18:05.783758+0000","last_fullsized":"2023-03-03T06:18:05.783758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766613+0000","last_change":"2023-03-03T06:18:05.377498+0000","last_active":"2023-03-03T06:18:06.766613+0000","last_peered":"2023-03-03T06:18:06.766613+0000","last_clean":"2023-03-03T06:18:06.766613+0000","last_became_active":"2023-03-03T06:18:05.377168+0000","last_became_peered":"2023-03-03T06:18:05.377168+0000","last_unstale":"2023-03-03T06:18:06.766613+0000","last_undegraded":"2023-03-03T06:18:06.766613+0000","last_fullsized":"2023-03-03T06:18:06.766613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:38.382151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761390+0000","last_change":"2023-03-03T06:18:05.779094+0000","last_active":"2023-03-03T06:18:06.761390+0000","last_peered":"2023-03-03T06:18:06.761390+0000","last_clean":"2023-03-03T06:18:06.761390+0000","last_became_active":"2023-03-03T06:18:05.778802+0000","last_became_peered":"2023-03-03T06:18:05.778802+0000","last_unstale":"2023-03-03T06:18:06.761390+0000","last_undegraded":"2023-03-03T06:18:06.761390+0000","last_fullsized":"2023-03-03T06:18:06.761390+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.759335+0000","last_change":"2023-03-03T06:18:05.376181+0000","last_active":"2023-03-03T06:18:05.759335+0000","last_peered":"2023-03-03T06:18:05.759335+0000","last_clean":"2023-03-03T06:18:05.759335+0000","last_became_active":"2023-03-03T06:18:05.375857+0000","last_became_peered":"2023-03-03T06:18:05.375857+0000","last_unstale":"2023-03-03T06:18:05.759335+0000","last_undegraded":"2023-03-03T06:18:05.759335+0000","last_fullsized":"2023-03-03T06:18:05.759335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767254+0000","last_change":"2023-03-03T06:18:05.775649+0000","last_active":"2023-03-03T06:18:06.767254+0000","last_peered":"2023-03-03T06:18:06.767254+0000","last_clean":"2023-03-03T06:18:06.767254+0000","last_became_active":"2023-03-03T06:18:05.775370+0000","last_became_peered":"2023-03-03T06:18:05.775370+0000","last_unstale":"2023-03-03T06:18:06.767254+0000","last_undegraded":"2023-03-03T06:18:06.767254+0000","last_fullsized":"2023-03-03T06:18:06.767254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766285+0000","last_change":"2023-03-03T06:18:05.377478+0000","last_active":"2023-03-03T06:18:06.766285+0000","last_peered":"2023-03-03T06:18:06.766285+0000","last_clean":"2023-03-03T06:18:06.766285+0000","last_became_active":"2023-03-03T06:18:05.377177+0000","last_became_peered":"2023-03-03T06:18:05.377177+0000","last_unstale":"2023-03-03T06:18:06.766285+0000","last_undegraded":"2023-03-03T06:18:06.766285+0000","last_fullsized":"2023-03-03T06:18:06.766285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:42.138865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773807+0000","last_change":"2023-03-03T06:18:05.771983+0000","last_active":"2023-03-03T06:18:05.773807+0000","last_peered":"2023-03-03T06:18:05.773807+0000","last_clean":"2023-03-03T06:18:05.773807+0000","last_became_active":"2023-03-03T06:18:05.771526+0000","last_became_peered":"2023-03-03T06:18:05.771526+0000","last_unstale":"2023-03-03T06:18:05.773807+0000","last_undegraded":"2023-03-03T06:18:05.773807+0000","last_fullsized":"2023-03-03T06:18:05.773807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783605+0000","last_change":"2023-03-03T06:18:05.379123+0000","last_active":"2023-03-03T06:18:05.783605+0000","last_peered":"2023-03-03T06:18:05.783605+0000","last_clean":"2023-03-03T06:18:05.783605+0000","last_became_active":"2023-03-03T06:18:05.378826+0000","last_became_peered":"2023-03-03T06:18:05.378826+0000","last_unstale":"2023-03-03T06:18:05.783605+0000","last_undegraded":"2023-03-03T06:18:05.783605+0000","last_fullsized":"2023-03-03T06:18:05.783605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761645+0000","last_change":"2023-03-03T06:18:05.766998+0000","last_active":"2023-03-03T06:18:06.761645+0000","last_peered":"2023-03-03T06:18:06.761645+0000","last_clean":"2023-03-03T06:18:06.761645+0000","last_became_active":"2023-03-03T06:18:05.764785+0000","last_became_peered":"2023-03-03T06:18:05.764785+0000","last_unstale":"2023-03-03T06:18:06.761645+0000","last_undegraded":"2023-03-03T06:18:06.761645+0000","last_fullsized":"2023-03-03T06:18:06.761645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767168+0000","last_change":"2023-03-03T06:18:05.377431+0000","last_active":"2023-03-03T06:18:06.767168+0000","last_peered":"2023-03-03T06:18:06.767168+0000","last_clean":"2023-03-03T06:18:06.767168+0000","last_became_active":"2023-03-03T06:18:05.377107+0000","last_became_peered":"2023-03-03T06:18:05.377107+0000","last_unstale":"2023-03-03T06:18:06.767168+0000","last_undegraded":"2023-03-03T06:18:06.767168+0000","last_fullsized":"2023-03-03T06:18:06.767168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766647+0000","last_change":"2023-03-03T06:18:05.774893+0000","last_active":"2023-03-03T06:18:06.766647+0000","last_peered":"2023-03-03T06:18:06.766647+0000","last_clean":"2023-03-03T06:18:06.766647+0000","last_became_active":"2023-03-03T06:18:05.774256+0000","last_became_peered":"2023-03-03T06:18:05.774256+0000","last_unstale":"2023-03-03T06:18:06.766647+0000","last_undegraded":"2023-03-03T06:18:06.766647+0000","last_fullsized":"2023-03-03T06:18:06.766647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766744+0000","last_change":"2023-03-03T06:18:05.377497+0000","last_active":"2023-03-03T06:18:06.766744+0000","last_peered":"2023-03-03T06:18:06.766744+0000","last_clean":"2023-03-03T06:18:06.766744+0000","last_became_active":"2023-03-03T06:18:05.377215+0000","last_became_peered":"2023-03-03T06:18:05.377215+0000","last_unstale":"2023-03-03T06:18:06.766744+0000","last_undegraded":"2023-03-03T06:18:06.766744+0000","last_fullsized":"2023-03-03T06:18:06.766744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783973+0000","last_change":"2023-03-03T06:18:05.781538+0000","last_active":"2023-03-03T06:18:05.783973+0000","last_peered":"2023-03-03T06:18:05.783973+0000","last_clean":"2023-03-03T06:18:05.783973+0000","last_became_active":"2023-03-03T06:18:05.781285+0000","last_became_peered":"2023-03-03T06:18:05.781285+0000","last_unstale":"2023-03-03T06:18:05.783973+0000","last_undegraded":"2023-03-03T06:18:05.783973+0000","last_fullsized":"2023-03-03T06:18:05.783973+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767086+0000","last_change":"2023-03-03T06:18:05.369584+0000","last_active":"2023-03-03T06:18:06.767086+0000","last_peered":"2023-03-03T06:18:06.767086+0000","last_clean":"2023-03-03T06:18:06.767086+0000","last_became_active":"2023-03-03T06:18:05.368904+0000","last_became_peered":"2023-03-03T06:18:05.368904+0000","last_unstale":"2023-03-03T06:18:06.767086+0000","last_undegraded":"2023-03-03T06:18:06.767086+0000","last_fullsized":"2023-03-03T06:18:06.767086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:23.861622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.786136+0000","last_change":"2023-03-03T06:18:05.779823+0000","last_active":"2023-03-03T06:18:05.786136+0000","last_peered":"2023-03-03T06:18:05.786136+0000","last_clean":"2023-03-03T06:18:05.786136+0000","last_became_active":"2023-03-03T06:18:05.779561+0000","last_became_peered":"2023-03-03T06:18:05.779561+0000","last_unstale":"2023-03-03T06:18:05.786136+0000","last_undegraded":"2023-03-03T06:18:05.786136+0000","last_fullsized":"2023-03-03T06:18:05.786136+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:54.968231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767196+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:06.767196+0000","last_peered":"2023-03-03T06:18:06.767196+0000","last_clean":"2023-03-03T06:18:06.767196+0000","last_became_active":"2023-03-03T06:18:05.372057+0000","last_became_peered":"2023-03-03T06:18:05.372057+0000","last_unstale":"2023-03-03T06:18:06.767196+0000","last_undegraded":"2023-03-03T06:18:06.767196+0000","last_fullsized":"2023-03-03T06:18:06.767196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782200+0000","last_change":"2023-03-03T06:18:05.780035+0000","last_active":"2023-03-03T06:18:05.782200+0000","last_peered":"2023-03-03T06:18:05.782200+0000","last_clean":"2023-03-03T06:18:05.782200+0000","last_became_active":"2023-03-03T06:18:05.779650+0000","last_became_peered":"2023-03-03T06:18:05.779650+0000","last_unstale":"2023-03-03T06:18:05.782200+0000","last_undegraded":"2023-03-03T06:18:05.782200+0000","last_fullsized":"2023-03-03T06:18:05.782200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761617+0000","last_change":"2023-03-03T06:18:05.379968+0000","last_active":"2023-03-03T06:18:06.761617+0000","last_peered":"2023-03-03T06:18:06.761617+0000","last_clean":"2023-03-03T06:18:06.761617+0000","last_became_active":"2023-03-03T06:18:05.379635+0000","last_became_peered":"2023-03-03T06:18:05.379635+0000","last_unstale":"2023-03-03T06:18:06.761617+0000","last_undegraded":"2023-03-03T06:18:06.761617+0000","last_fullsized":"2023-03-03T06:18:06.761617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.777783+0000","last_change":"2023-03-03T06:18:05.768972+0000","last_active":"2023-03-03T06:18:05.777783+0000","last_peered":"2023-03-03T06:18:05.777783+0000","last_clean":"2023-03-03T06:18:05.777783+0000","last_became_active":"2023-03-03T06:18:05.768647+0000","last_became_peered":"2023-03-03T06:18:05.768647+0000","last_unstale":"2023-03-03T06:18:05.777783+0000","last_undegraded":"2023-03-03T06:18:05.777783+0000","last_fullsized":"2023-03-03T06:18:05.777783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:15:35.033362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.765000+0000","last_change":"2023-03-03T06:18:05.378643+0000","last_active":"2023-03-03T06:18:05.765000+0000","last_peered":"2023-03-03T06:18:05.765000+0000","last_clean":"2023-03-03T06:18:05.765000+0000","last_became_active":"2023-03-03T06:18:05.378333+0000","last_became_peered":"2023-03-03T06:18:05.378333+0000","last_unstale":"2023-03-03T06:18:05.765000+0000","last_undegraded":"2023-03-03T06:18:05.765000+0000","last_fullsized":"2023-03-03T06:18:05.765000+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780860+0000","last_change":"2023-03-03T06:18:05.774576+0000","last_active":"2023-03-03T06:18:05.780860+0000","last_peered":"2023-03-03T06:18:05.780860+0000","last_clean":"2023-03-03T06:18:05.780860+0000","last_became_active":"2023-03-03T06:18:05.774278+0000","last_became_peered":"2023-03-03T06:18:05.774278+0000","last_unstale":"2023-03-03T06:18:05.780860+0000","last_undegraded":"2023-03-03T06:18:05.780860+0000","last_fullsized":"2023-03-03T06:18:05.780860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:20.212207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.767475+0000","last_change":"2023-03-03T06:18:05.374483+0000","last_active":"2023-03-03T06:18:05.767475+0000","last_peered":"2023-03-03T06:18:05.767475+0000","last_clean":"2023-03-03T06:18:05.767475+0000","last_became_active":"2023-03-03T06:18:05.374224+0000","last_became_peered":"2023-03-03T06:18:05.374224+0000","last_unstale":"2023-03-03T06:18:05.767475+0000","last_undegraded":"2023-03-03T06:18:05.767475+0000","last_fullsized":"2023-03-03T06:18:05.767475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783938+0000","last_change":"2023-03-03T06:18:05.779652+0000","last_active":"2023-03-03T06:18:05.783938+0000","last_peered":"2023-03-03T06:18:05.783938+0000","last_clean":"2023-03-03T06:18:05.783938+0000","last_became_active":"2023-03-03T06:18:05.778557+0000","last_became_peered":"2023-03-03T06:18:05.778557+0000","last_unstale":"2023-03-03T06:18:05.783938+0000","last_undegraded":"2023-03-03T06:18:05.783938+0000","last_fullsized":"2023-03-03T06:18:05.783938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761087+0000","last_change":"2023-03-03T06:18:05.373177+0000","last_active":"2023-03-03T06:18:06.761087+0000","last_peered":"2023-03-03T06:18:06.761087+0000","last_clean":"2023-03-03T06:18:06.761087+0000","last_became_active":"2023-03-03T06:18:05.372879+0000","last_became_peered":"2023-03-03T06:18:05.372879+0000","last_unstale":"2023-03-03T06:18:06.761087+0000","last_undegraded":"2023-03-03T06:18:06.761087+0000","last_fullsized":"2023-03-03T06:18:06.761087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782728+0000","last_change":"2023-03-03T06:18:05.778960+0000","last_active":"2023-03-03T06:18:05.782728+0000","last_peered":"2023-03-03T06:18:05.782728+0000","last_clean":"2023-03-03T06:18:05.782728+0000","last_became_active":"2023-03-03T06:18:05.777570+0000","last_became_peered":"2023-03-03T06:18:05.777570+0000","last_unstale":"2023-03-03T06:18:05.782728+0000","last_undegraded":"2023-03-03T06:18:05.782728+0000","last_fullsized":"2023-03-03T06:18:05.782728+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.357551+0000","last_change":"2023-03-03T06:18:00.701360+0000","last_active":"2023-03-03T06:18:05.357551+0000","last_peered":"2023-03-03T06:18:05.357551+0000","last_clean":"2023-03-03T06:18:05.357551+0000","last_became_active":"2023-03-03T06:18:00.701042+0000","last_became_peered":"2023-03-03T06:18:00.701042+0000","last_unstale":"2023-03-03T06:18:05.357551+0000","last_undegraded":"2023-03-03T06:18:05.357551+0000","last_fullsized":"2023-03-03T06:18:05.357551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:10.253889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767335+0000","last_change":"2023-03-03T06:18:05.372375+0000","last_active":"2023-03-03T06:18:06.767335+0000","last_peered":"2023-03-03T06:18:06.767335+0000","last_clean":"2023-03-03T06:18:06.767335+0000","last_became_active":"2023-03-03T06:18:05.371680+0000","last_became_peered":"2023-03-03T06:18:05.371680+0000","last_unstale":"2023-03-03T06:18:06.767335+0000","last_undegraded":"2023-03-03T06:18:06.767335+0000","last_fullsized":"2023-03-03T06:18:06.767335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782775+0000","last_change":"2023-03-03T06:18:05.778931+0000","last_active":"2023-03-03T06:18:05.782775+0000","last_peered":"2023-03-03T06:18:05.782775+0000","last_clean":"2023-03-03T06:18:05.782775+0000","last_became_active":"2023-03-03T06:18:05.778723+0000","last_became_peered":"2023-03-03T06:18:05.778723+0000","last_unstale":"2023-03-03T06:18:05.782775+0000","last_undegraded":"2023-03-03T06:18:05.782775+0000","last_fullsized":"2023-03-03T06:18:05.782775+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.764427+0000","last_change":"2023-03-03T06:18:00.351410+0000","last_active":"2023-03-03T06:18:05.764427+0000","last_peered":"2023-03-03T06:18:05.764427+0000","last_clean":"2023-03-03T06:18:05.764427+0000","last_became_active":"2023-03-03T06:18:00.351036+0000","last_became_peered":"2023-03-03T06:18:00.351036+0000","last_unstale":"2023-03-03T06:18:05.764427+0000","last_undegraded":"2023-03-03T06:18:05.764427+0000","last_fullsized":"2023-03-03T06:18:05.764427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.763797+0000","last_change":"2023-03-03T06:18:05.378884+0000","last_active":"2023-03-03T06:18:05.763797+0000","last_peered":"2023-03-03T06:18:05.763797+0000","last_clean":"2023-03-03T06:18:05.763797+0000","last_became_active":"2023-03-03T06:18:05.378597+0000","last_became_peered":"2023-03-03T06:18:05.378597+0000","last_unstale":"2023-03-03T06:18:05.763797+0000","last_undegraded":"2023-03-03T06:18:05.763797+0000","last_fullsized":"2023-03-03T06:18:05.763797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780959+0000","last_change":"2023-03-03T06:18:05.777398+0000","last_active":"2023-03-03T06:18:05.780959+0000","last_peered":"2023-03-03T06:18:05.780959+0000","last_clean":"2023-03-03T06:18:05.780959+0000","last_became_active":"2023-03-03T06:18:05.777065+0000","last_became_peered":"2023-03-03T06:18:05.777065+0000","last_unstale":"2023-03-03T06:18:05.780959+0000","last_undegraded":"2023-03-03T06:18:05.780959+0000","last_fullsized":"2023-03-03T06:18:05.780959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.365191+0000","last_change":"2023-03-03T06:18:00.351233+0000","last_active":"2023-03-03T06:18:05.365191+0000","last_peered":"2023-03-03T06:18:05.365191+0000","last_clean":"2023-03-03T06:18:05.365191+0000","last_became_active":"2023-03-03T06:18:00.350944+0000","last_became_peered":"2023-03-03T06:18:00.350944+0000","last_unstale":"2023-03-03T06:18:05.365191+0000","last_undegraded":"2023-03-03T06:18:05.365191+0000","last_fullsized":"2023-03-03T06:18:05.365191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:08:23.569811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.770017+0000","last_change":"2023-03-03T06:18:05.374876+0000","last_active":"2023-03-03T06:18:05.770017+0000","last_peered":"2023-03-03T06:18:05.770017+0000","last_clean":"2023-03-03T06:18:05.770017+0000","last_became_active":"2023-03-03T06:18:05.374266+0000","last_became_peered":"2023-03-03T06:18:05.374266+0000","last_unstale":"2023-03-03T06:18:05.770017+0000","last_undegraded":"2023-03-03T06:18:05.770017+0000","last_fullsized":"2023-03-03T06:18:05.770017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782760+0000","last_change":"2023-03-03T06:18:05.779139+0000","last_active":"2023-03-03T06:18:05.782760+0000","last_peered":"2023-03-03T06:18:05.782760+0000","last_clean":"2023-03-03T06:18:05.782760+0000","last_became_active":"2023-03-03T06:18:05.777315+0000","last_became_peered":"2023-03-03T06:18:05.777315+0000","last_unstale":"2023-03-03T06:18:05.782760+0000","last_undegraded":"2023-03-03T06:18:05.782760+0000","last_fullsized":"2023-03-03T06:18:05.782760+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.762243+0000","last_change":"2023-03-03T06:18:00.351257+0000","last_active":"2023-03-03T06:18:05.762243+0000","last_peered":"2023-03-03T06:18:05.762243+0000","last_clean":"2023-03-03T06:18:05.762243+0000","last_became_active":"2023-03-03T06:18:00.350933+0000","last_became_peered":"2023-03-03T06:18:00.350933+0000","last_unstale":"2023-03-03T06:18:05.762243+0000","last_undegraded":"2023-03-03T06:18:05.762243+0000","last_fullsized":"2023-03-03T06:18:05.762243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758385+0000","last_change":"2023-03-03T06:17:54.307666+0000","last_active":"2023-03-03T06:18:05.758385+0000","last_peered":"2023-03-03T06:18:05.758385+0000","last_clean":"2023-03-03T06:18:05.758385+0000","last_became_active":"2023-03-03T06:17:54.306552+0000","last_became_peered":"2023-03-03T06:17:54.306552+0000","last_unstale":"2023-03-03T06:18:05.758385+0000","last_undegraded":"2023-03-03T06:18:05.758385+0000","last_fullsized":"2023-03-03T06:18:05.758385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_clean_scrub_stamp":"2023-03-03T06:17:53.091715+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:56:41.267904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761267+0000","last_change":"2023-03-03T06:18:05.366688+0000","last_active":"2023-03-03T06:18:06.761267+0000","last_peered":"2023-03-03T06:18:06.761267+0000","last_clean":"2023-03-03T06:18:06.761267+0000","last_became_active":"2023-03-03T06:18:05.366412+0000","last_became_peered":"2023-03-03T06:18:05.366412+0000","last_unstale":"2023-03-03T06:18:06.761267+0000","last_undegraded":"2023-03-03T06:18:06.761267+0000","last_fullsized":"2023-03-03T06:18:06.761267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.786219+0000","last_change":"2023-03-03T06:18:05.781486+0000","last_active":"2023-03-03T06:18:05.786219+0000","last_peered":"2023-03-03T06:18:05.786219+0000","last_clean":"2023-03-03T06:18:05.786219+0000","last_became_active":"2023-03-03T06:18:05.781220+0000","last_became_peered":"2023-03-03T06:18:05.781220+0000","last_unstale":"2023-03-03T06:18:05.786219+0000","last_undegraded":"2023-03-03T06:18:05.786219+0000","last_fullsized":"2023-03-03T06:18:05.786219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:39.387687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758932+0000","last_change":"2023-03-03T06:18:00.345967+0000","last_active":"2023-03-03T06:18:05.758932+0000","last_peered":"2023-03-03T06:18:05.758932+0000","last_clean":"2023-03-03T06:18:05.758932+0000","last_became_active":"2023-03-03T06:18:00.345666+0000","last_became_peered":"2023-03-03T06:18:00.345666+0000","last_unstale":"2023-03-03T06:18:05.758932+0000","last_undegraded":"2023-03-03T06:18:05.758932+0000","last_fullsized":"2023-03-03T06:18:05.758932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.760333+0000","last_change":"2023-03-03T06:18:05.379214+0000","last_active":"2023-03-03T06:18:05.760333+0000","last_peered":"2023-03-03T06:18:05.760333+0000","last_clean":"2023-03-03T06:18:05.760333+0000","last_became_active":"2023-03-03T06:18:05.378867+0000","last_became_peered":"2023-03-03T06:18:05.378867+0000","last_unstale":"2023-03-03T06:18:05.760333+0000","last_undegraded":"2023-03-03T06:18:05.760333+0000","last_fullsized":"2023-03-03T06:18:05.760333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783637+0000","last_change":"2023-03-03T06:18:05.782275+0000","last_active":"2023-03-03T06:18:05.783637+0000","last_peered":"2023-03-03T06:18:05.783637+0000","last_clean":"2023-03-03T06:18:05.783637+0000","last_became_active":"2023-03-03T06:18:05.781968+0000","last_became_peered":"2023-03-03T06:18:05.781968+0000","last_unstale":"2023-03-03T06:18:05.783637+0000","last_undegraded":"2023-03-03T06:18:05.783637+0000","last_fullsized":"2023-03-03T06:18:05.783637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.759159+0000","last_change":"2023-03-03T06:18:00.701652+0000","last_active":"2023-03-03T06:18:05.759159+0000","last_peered":"2023-03-03T06:18:05.759159+0000","last_clean":"2023-03-03T06:18:05.759159+0000","last_became_active":"2023-03-03T06:18:00.701382+0000","last_became_peered":"2023-03-03T06:18:00.701382+0000","last_unstale":"2023-03-03T06:18:05.759159+0000","last_undegraded":"2023-03-03T06:18:05.759159+0000","last_fullsized":"2023-03-03T06:18:05.759159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:50:16.060640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.775494+0000","last_change":"2023-03-03T06:18:05.374785+0000","last_active":"2023-03-03T06:18:05.775494+0000","last_peered":"2023-03-03T06:18:05.775494+0000","last_clean":"2023-03-03T06:18:05.775494+0000","last_became_active":"2023-03-03T06:18:05.374322+0000","last_became_peered":"2023-03-03T06:18:05.374322+0000","last_unstale":"2023-03-03T06:18:05.775494+0000","last_undegraded":"2023-03-03T06:18:05.775494+0000","last_fullsized":"2023-03-03T06:18:05.775494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.779863+0000","last_change":"2023-03-03T06:18:05.775603+0000","last_active":"2023-03-03T06:18:05.779863+0000","last_peered":"2023-03-03T06:18:05.779863+0000","last_clean":"2023-03-03T06:18:05.779863+0000","last_became_active":"2023-03-03T06:18:05.775350+0000","last_became_peered":"2023-03-03T06:18:05.775350+0000","last_unstale":"2023-03-03T06:18:05.779863+0000","last_undegraded":"2023-03-03T06:18:05.779863+0000","last_fullsized":"2023-03-03T06:18:05.779863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761609+0000","last_change":"2023-03-03T06:18:05.371388+0000","last_active":"2023-03-03T06:18:06.761609+0000","last_peered":"2023-03-03T06:18:06.761609+0000","last_clean":"2023-03-03T06:18:06.761609+0000","last_became_active":"2023-03-03T06:18:05.371026+0000","last_became_peered":"2023-03-03T06:18:05.371026+0000","last_unstale":"2023-03-03T06:18:06.761609+0000","last_undegraded":"2023-03-03T06:18:06.761609+0000","last_fullsized":"2023-03-03T06:18:06.761609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.357510+0000","last_change":"2023-03-03T06:18:00.349337+0000","last_active":"2023-03-03T06:18:05.357510+0000","last_peered":"2023-03-03T06:18:05.357510+0000","last_clean":"2023-03-03T06:18:05.357510+0000","last_became_active":"2023-03-03T06:18:00.349101+0000","last_became_peered":"2023-03-03T06:18:00.349101+0000","last_unstale":"2023-03-03T06:18:05.357510+0000","last_undegraded":"2023-03-03T06:18:05.357510+0000","last_fullsized":"2023-03-03T06:18:05.357510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761022+0000","last_change":"2023-03-03T06:18:05.762420+0000","last_active":"2023-03-03T06:18:06.761022+0000","last_peered":"2023-03-03T06:18:06.761022+0000","last_clean":"2023-03-03T06:18:06.761022+0000","last_became_active":"2023-03-03T06:18:05.760911+0000","last_became_peered":"2023-03-03T06:18:05.760911+0000","last_unstale":"2023-03-03T06:18:06.761022+0000","last_undegraded":"2023-03-03T06:18:06.761022+0000","last_fullsized":"2023-03-03T06:18:06.761022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.365729+0000","last_change":"2023-03-03T06:18:00.352230+0000","last_active":"2023-03-03T06:18:05.365729+0000","last_peered":"2023-03-03T06:18:05.365729+0000","last_clean":"2023-03-03T06:18:05.365729+0000","last_became_active":"2023-03-03T06:18:00.351881+0000","last_became_peered":"2023-03-03T06:18:00.351881+0000","last_unstale":"2023-03-03T06:18:05.365729+0000","last_undegraded":"2023-03-03T06:18:05.365729+0000","last_fullsized":"2023-03-03T06:18:05.365729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:18:24.984007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.760860+0000","last_change":"2023-03-03T06:18:05.382451+0000","last_active":"2023-03-03T06:18:05.760860+0000","last_peered":"2023-03-03T06:18:05.760860+0000","last_clean":"2023-03-03T06:18:05.760860+0000","last_became_active":"2023-03-03T06:18:05.382112+0000","last_became_peered":"2023-03-03T06:18:05.382112+0000","last_unstale":"2023-03-03T06:18:05.760860+0000","last_undegraded":"2023-03-03T06:18:05.760860+0000","last_fullsized":"2023-03-03T06:18:05.760860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783537+0000","last_change":"2023-03-03T06:18:05.777771+0000","last_active":"2023-03-03T06:18:05.783537+0000","last_peered":"2023-03-03T06:18:05.783537+0000","last_clean":"2023-03-03T06:18:05.783537+0000","last_became_active":"2023-03-03T06:18:05.777457+0000","last_became_peered":"2023-03-03T06:18:05.777457+0000","last_unstale":"2023-03-03T06:18:05.783537+0000","last_undegraded":"2023-03-03T06:18:05.783537+0000","last_fullsized":"2023-03-03T06:18:05.783537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:24.867158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.771241+0000","last_change":"2023-03-03T06:18:05.382340+0000","last_active":"2023-03-03T06:18:05.771241+0000","last_peered":"2023-03-03T06:18:05.771241+0000","last_clean":"2023-03-03T06:18:05.771241+0000","last_became_active":"2023-03-03T06:18:05.382051+0000","last_became_peered":"2023-03-03T06:18:05.382051+0000","last_unstale":"2023-03-03T06:18:05.771241+0000","last_undegraded":"2023-03-03T06:18:05.771241+0000","last_fullsized":"2023-03-03T06:18:05.771241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783095+0000","last_change":"2023-03-03T06:18:05.778551+0000","last_active":"2023-03-03T06:18:05.783095+0000","last_peered":"2023-03-03T06:18:05.783095+0000","last_clean":"2023-03-03T06:18:05.783095+0000","last_became_active":"2023-03-03T06:18:05.778166+0000","last_became_peered":"2023-03-03T06:18:05.778166+0000","last_unstale":"2023-03-03T06:18:05.783095+0000","last_undegraded":"2023-03-03T06:18:05.783095+0000","last_fullsized":"2023-03-03T06:18:05.783095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:39.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766897+0000","last_change":"2023-03-03T06:18:05.371014+0000","last_active":"2023-03-03T06:18:06.766897+0000","last_peered":"2023-03-03T06:18:06.766897+0000","last_clean":"2023-03-03T06:18:06.766897+0000","last_became_active":"2023-03-03T06:18:05.370730+0000","last_became_peered":"2023-03-03T06:18:05.370730+0000","last_unstale":"2023-03-03T06:18:06.766897+0000","last_undegraded":"2023-03-03T06:18:06.766897+0000","last_fullsized":"2023-03-03T06:18:06.766897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780783+0000","last_change":"2023-03-03T06:18:05.772599+0000","last_active":"2023-03-03T06:18:05.780783+0000","last_peered":"2023-03-03T06:18:05.780783+0000","last_clean":"2023-03-03T06:18:05.780783+0000","last_became_active":"2023-03-03T06:18:05.772305+0000","last_became_peered":"2023-03-03T06:18:05.772305+0000","last_unstale":"2023-03-03T06:18:05.780783+0000","last_undegraded":"2023-03-03T06:18:05.780783+0000","last_fullsized":"2023-03-03T06:18:05.780783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:14.254976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767411+0000","last_change":"2023-03-03T06:18:05.374466+0000","last_active":"2023-03-03T06:18:06.767411+0000","last_peered":"2023-03-03T06:18:06.767411+0000","last_clean":"2023-03-03T06:18:06.767411+0000","last_became_active":"2023-03-03T06:18:05.374194+0000","last_became_peered":"2023-03-03T06:18:05.374194+0000","last_unstale":"2023-03-03T06:18:06.767411+0000","last_undegraded":"2023-03-03T06:18:06.767411+0000","last_fullsized":"2023-03-03T06:18:06.767411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781812+0000","last_change":"2023-03-03T06:18:05.773340+0000","last_active":"2023-03-03T06:18:05.781812+0000","last_peered":"2023-03-03T06:18:05.781812+0000","last_clean":"2023-03-03T06:18:05.781812+0000","last_became_active":"2023-03-03T06:18:05.772391+0000","last_became_peered":"2023-03-03T06:18:05.772391+0000","last_unstale":"2023-03-03T06:18:05.781812+0000","last_undegraded":"2023-03-03T06:18:05.781812+0000","last_fullsized":"2023-03-03T06:18:05.781812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.762090+0000","last_change":"2023-03-03T06:18:05.369911+0000","last_active":"2023-03-03T06:18:06.762090+0000","last_peered":"2023-03-03T06:18:06.762090+0000","last_clean":"2023-03-03T06:18:06.762090+0000","last_became_active":"2023-03-03T06:18:05.369605+0000","last_became_peered":"2023-03-03T06:18:05.369605+0000","last_unstale":"2023-03-03T06:18:06.762090+0000","last_undegraded":"2023-03-03T06:18:06.762090+0000","last_fullsized":"2023-03-03T06:18:06.762090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761116+0000","last_change":"2023-03-03T06:18:05.766192+0000","last_active":"2023-03-03T06:18:06.761116+0000","last_peered":"2023-03-03T06:18:06.761116+0000","last_clean":"2023-03-03T06:18:06.761116+0000","last_became_active":"2023-03-03T06:18:05.764744+0000","last_became_peered":"2023-03-03T06:18:05.764744+0000","last_unstale":"2023-03-03T06:18:06.761116+0000","last_undegraded":"2023-03-03T06:18:06.761116+0000","last_fullsized":"2023-03-03T06:18:06.761116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.776193+0000","last_change":"2023-03-03T06:18:05.378275+0000","last_active":"2023-03-03T06:18:05.776193+0000","last_peered":"2023-03-03T06:18:05.776193+0000","last_clean":"2023-03-03T06:18:05.776193+0000","last_became_active":"2023-03-03T06:18:05.377803+0000","last_became_peered":"2023-03-03T06:18:05.377803+0000","last_unstale":"2023-03-03T06:18:05.776193+0000","last_undegraded":"2023-03-03T06:18:05.776193+0000","last_fullsized":"2023-03-03T06:18:05.776193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780746+0000","last_change":"2023-03-03T06:18:05.770801+0000","last_active":"2023-03-03T06:18:05.780746+0000","last_peered":"2023-03-03T06:18:05.780746+0000","last_clean":"2023-03-03T06:18:05.780746+0000","last_became_active":"2023-03-03T06:18:05.770500+0000","last_became_peered":"2023-03-03T06:18:05.770500+0000","last_unstale":"2023-03-03T06:18:05.780746+0000","last_undegraded":"2023-03-03T06:18:05.780746+0000","last_fullsized":"2023-03-03T06:18:05.780746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783309+0000","last_change":"2023-03-03T06:18:05.776954+0000","last_active":"2023-03-03T06:18:05.783309+0000","last_peered":"2023-03-03T06:18:05.783309+0000","last_clean":"2023-03-03T06:18:05.783309+0000","last_became_active":"2023-03-03T06:18:05.776665+0000","last_became_peered":"2023-03-03T06:18:05.776665+0000","last_unstale":"2023-03-03T06:18:05.783309+0000","last_undegraded":"2023-03-03T06:18:05.783309+0000","last_fullsized":"2023-03-03T06:18:05.783309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:46.917574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761055+0000","last_change":"2023-03-03T06:18:05.780029+0000","last_active":"2023-03-03T06:18:06.761055+0000","last_peered":"2023-03-03T06:18:06.761055+0000","last_clean":"2023-03-03T06:18:06.761055+0000","last_became_active":"2023-03-03T06:18:05.779805+0000","last_became_peered":"2023-03-03T06:18:05.779805+0000","last_unstale":"2023-03-03T06:18:06.761055+0000","last_undegraded":"2023-03-03T06:18:06.761055+0000","last_fullsized":"2023-03-03T06:18:06.761055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.784005+0000","last_change":"2023-03-03T06:18:05.781886+0000","last_active":"2023-03-03T06:18:05.784005+0000","last_peered":"2023-03-03T06:18:05.784005+0000","last_clean":"2023-03-03T06:18:05.784005+0000","last_became_active":"2023-03-03T06:18:05.780587+0000","last_became_peered":"2023-03-03T06:18:05.780587+0000","last_unstale":"2023-03-03T06:18:05.784005+0000","last_undegraded":"2023-03-03T06:18:05.784005+0000","last_fullsized":"2023-03-03T06:18:05.784005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:02.129741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761713+0000","last_change":"2023-03-03T06:18:05.760339+0000","last_active":"2023-03-03T06:18:06.761713+0000","last_peered":"2023-03-03T06:18:06.761713+0000","last_clean":"2023-03-03T06:18:06.761713+0000","last_became_active":"2023-03-03T06:18:05.759652+0000","last_became_peered":"2023-03-03T06:18:05.759652+0000","last_unstale":"2023-03-03T06:18:06.761713+0000","last_undegraded":"2023-03-03T06:18:06.761713+0000","last_fullsized":"2023-03-03T06:18:06.761713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780979+0000","last_change":"2023-03-03T06:18:05.777608+0000","last_active":"2023-03-03T06:18:05.780979+0000","last_peered":"2023-03-03T06:18:05.780979+0000","last_clean":"2023-03-03T06:18:05.780979+0000","last_became_active":"2023-03-03T06:18:05.777310+0000","last_became_peered":"2023-03-03T06:18:05.777310+0000","last_unstale":"2023-03-03T06:18:05.780979+0000","last_undegraded":"2023-03-03T06:18:05.780979+0000","last_fullsized":"2023-03-03T06:18:05.780979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9096,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362744,"statfs":{"total":96636764160,"available":96627449856,"internally_reserved":0,"allocated":851968,"data_stored":675078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7976,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363864,"statfs":{"total":96636764160,"available":96628596736,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7976,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363864,"statfs":{"total":96636764160,"available":96628596736,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9072,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362768,"statfs":{"total":96636764160,"available":96627474432,"internally_reserved":0,"allocated":827392,"data_stored":673880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7912,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363928,"statfs":{"total":96636764160,"available":96628662272,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8044,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363796,"statfs":{"total":96636764160,"available":96628527104,"internally_reserved":0,"allocated":233472,"data_stored":83036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8060,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363780,"statfs":{"total":96636764160,"available":96628510720,"internally_reserved":0,"allocated":249856,"data_stored":84550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6040000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7050000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":83582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T06:18:09.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T06:18:09.979 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.977+0000 7f11f4c50700 1 Processor -- start 2023-03-03T06:18:09.979 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.977+0000 7f11f4c50700 1 -- start start 2023-03-03T06:18:09.982 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11f4c50700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11f0145970 0x7f11f00bd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.982 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11f4c50700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 0x7f11f014ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.982 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11f4c50700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f006dd10 con 0x7f11f00bdd00 2023-03-03T06:18:09.982 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11f4c50700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f00c15a0 con 0x7f11f0145970 2023-03-03T06:18:09.982 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11f4c50700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00bc380 con 0x7f11f00bcf00 2023-03-03T06:18:09.983 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11eed9d700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f11f00bdd00 0x7f11f006a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53230/0 (socket says 172.21.15.90:53230) 2023-03-03T06:18:09.983 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11eed9d700 1 -- 172.21.15.90:0/1128221190 learned_addr learned my addr 172.21.15.90:0/1128221190 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:09.983 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11edd9b700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 0x7f11f014ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.983 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657523247 0 0) 0x7f11f006dd10 con 0x7f11f00bdd00 2023-03-03T06:18:09.983 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d80034d0 con 0x7f11f00bdd00 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ee59c700 1 --2- 172.21.15.90:0/1128221190 >> v2:172.21.15.121:3300/0 conn(0x7f11f0145970 0x7f11f00bd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711336442 0 0) 0x7f11d80034d0 con 0x7f11f00bdd00 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 msgr2=0x7f11f014ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 0x7f11f014ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 >> v2:172.21.15.121:3300/0 conn(0x7f11f0145970 msgr2=0x7f11f00bd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 --2- 172.21.15.90:0/1128221190 >> v2:172.21.15.121:3300/0 conn(0x7f11f0145970 0x7f11f00bd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f00c1300 con 0x7f11f00bdd00 2023-03-03T06:18:09.984 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f11e80019f0 con 0x7f11f00bdd00 2023-03-03T06:18:09.985 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.978+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 >> v1:172.21.15.90:6789/0 conn(0x7f11f00bdd00 legacy=0x7f11f006a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.985 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 0x7f11d8003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.985 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11d800a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.985 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d800af20 0x7f11d800d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.985 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00c15a0 con 0x7f11d8004110 2023-03-03T06:18:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d800dc60 con 0x7f11d800af20 2023-03-03T06:18:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d800de20 con 0x7f11f00bcf00 2023-03-03T06:18:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11edd9b700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11d800a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ee59c700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 0x7f11d8003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11eed9d700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d800af20 0x7f11d800d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11edd9b700 1 -- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 msgr2=0x7f11d8003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11edd9b700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f00bcf00 0x7f11d8003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11edd9b700 1 -- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d800af20 msgr2=0x7f11d800d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11edd9b700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d800af20 0x7f11d800d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11edd9b700 1 -- 172.21.15.90:0/1128221190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d80034d0 con 0x7f11d8004110 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11edd9b700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11d800a8f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009b60 tx=0x7f11d8003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976e59b7bdd17e10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e000e6c0 con 0x7f11d8004110 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e000e820 con 0x7f11d8004110 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.979+0000 7f11ed59a700 1 -- 172.21.15.90:0/1128221190 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e000eb90 con 0x7f11d8004110 2023-03-03T06:18:09.987 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.980+0000 7f11f4c50700 1 -- 172.21.15.90:0/1128221190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 msgr2=0x7f11d800a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.988 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.980+0000 7f11f4c50700 1 --2- 172.21.15.90:0/1128221190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11d800a8f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009b60 tx=0x7f11d8003780 comp rx=0 tx=0).stop 2023-03-03T06:18:09.988 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.980+0000 7f11eed9d700 1 -- 172.21.15.90:0/1128221190 reap_dead start 2023-03-03T06:18:09.988 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.980+0000 7f11f4c50700 1 -- 172.21.15.90:0/1128221190 shutdown_connections 2023-03-03T06:18:09.988 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.980+0000 7f11f4c50700 1 -- 172.21.15.90:0/1128221190 >> 172.21.15.90:0/1128221190 conn(0x7f11f00c1900 msgr2=0x7f11f00c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:09.988 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.980+0000 7f11f4c50700 1 -- 172.21.15.90:0/1128221190 shutdown_connections 2023-03-03T06:18:09.988 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.980+0000 7f11f4c50700 1 -- 172.21.15.90:0/1128221190 wait complete. 2023-03-03T06:18:09.990 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.981+0000 7f11f4c50700 1 Processor -- start 2023-03-03T06:18:09.991 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.981+0000 7f11f4c50700 1 -- start start 2023-03-03T06:18:09.991 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11f0071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.991 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f01d4610 0x7f11f01e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.991 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f01e5020 0x7f11f013f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.991 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00a3460 con 0x7f11d8004110 2023-03-03T06:18:09.991 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f01d5060 con 0x7f11f01e5020 2023-03-03T06:18:09.991 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0071640 con 0x7f11f01d4610 2023-03-03T06:18:09.992 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11f0071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.992 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11f0071100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34032/0 (socket says 172.21.15.90:34032) 2023-03-03T06:18:09.992 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 -- 172.21.15.90:0/999007863 learned_addr learned my addr 172.21.15.90:0/999007863 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:09.992 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11edd9b700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f01d4610 0x7f11f01e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.992 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11eed9d700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f01e5020 0x7f11f013f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.992 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 -- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f01d4610 msgr2=0x7f11f01e4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.992 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f01d4610 0x7f11f01e4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.993 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 -- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f01e5020 msgr2=0x7f11f013f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:09.993 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f01e5020 0x7f11f013f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:09.993 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 -- 172.21.15.90:0/999007863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f00a0e60 con 0x7f11d8004110 2023-03-03T06:18:09.993 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11eed9d700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f01e5020 0x7f11f013f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:09.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11ee59c700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11f0071100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f11e4011a00 tx=0x7f11e4011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11deffd700 1 -- 172.21.15.90:0/999007863 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e400f940 con 0x7f11d8004110 2023-03-03T06:18:09.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11deffd700 1 -- 172.21.15.90:0/999007863 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e400fac0 con 0x7f11d8004110 2023-03-03T06:18:09.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f006da30 con 0x7f11d8004110 2023-03-03T06:18:09.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11deffd700 1 -- 172.21.15.90:0/999007863 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e400feb0 con 0x7f11d8004110 2023-03-03T06:18:09.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.982+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f00a1140 con 0x7f11d8004110 2023-03-03T06:18:09.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.983+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f00a16c0 con 0x7f11d8004110 2023-03-03T06:18:09.995 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.984+0000 7f11deffd700 1 -- 172.21.15.90:0/999007863 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11e4014440 con 0x7f11d8004110 2023-03-03T06:18:09.995 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.984+0000 7f11deffd700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f11d0068320 0x7f11d006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:09.995 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.984+0000 7f11edd9b700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f11d0068320 0x7f11d006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:09.995 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.984+0000 7f11deffd700 1 -- 172.21.15.90:0/999007863 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f11e4086480 con 0x7f11d8004110 2023-03-03T06:18:09.995 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.985+0000 7f11edd9b700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f11d0068320 0x7f11d006a7d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f11e00090b0 tx=0x7f11e001f040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:09.995 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:09.988+0000 7f11deffd700 1 -- 172.21.15.90:0/999007863 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11f00a16c0 con 0x7f11d8004110 2023-03-03T06:18:10.168 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.166+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 --> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11f00a16c0 con 0x7f11d0068320 2023-03-03T06:18:10.175 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.174+0000 7f11deffd700 1 -- 172.21.15.90:0/999007863 <== mgr.4108 v2:172.21.15.90:6832/96475 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286674 (secure 0 0 0) 0x7f11f00a16c0 con 0x7f11d0068320 2023-03-03T06:18:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:10.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f11d0068320 msgr2=0x7f11d006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f11d0068320 0x7f11d006a7d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f11e00090b0 tx=0x7f11e001f040 comp rx=0 tx=0).stop 2023-03-03T06:18:10.180 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 msgr2=0x7f11f0071100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11f0071100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f11e4011a00 tx=0x7f11e4011d10 comp rx=0 tx=0).stop 2023-03-03T06:18:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 shutdown_connections 2023-03-03T06:18:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11f01e5020 0x7f11f013f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f11d0068320 0x7f11d006a7d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d8004110 0x7f11f0071100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 --2- 172.21.15.90:0/999007863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11f01d4610 0x7f11f01e4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.181 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 >> 172.21.15.90:0/999007863 conn(0x7f11f00c1900 msgr2=0x7f11f00c04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 shutdown_connections 2023-03-03T06:18:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.177+0000 7f11f4c50700 1 -- 172.21.15.90:0/999007863 wait complete. 2023-03-03T06:18:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-03-03T06:18:10.190 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-03-03T06:18:10.075032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66120,"kb_used_data":3016,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":754908600,"statfs":{"total":773094113280,"available":773026406400,"internally_reserved":0,"allocated":3088384,"data_stored":1848964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8],"upper_bound":8},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.725183"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773776+0000","last_change":"2023-03-03T06:18:05.768223+0000","last_active":"2023-03-03T06:18:05.773776+0000","last_peered":"2023-03-03T06:18:05.773776+0000","last_clean":"2023-03-03T06:18:05.773776+0000","last_became_active":"2023-03-03T06:18:05.767943+0000","last_became_peered":"2023-03-03T06:18:05.767943+0000","last_unstale":"2023-03-03T06:18:05.773776+0000","last_undegraded":"2023-03-03T06:18:05.773776+0000","last_fullsized":"2023-03-03T06:18:05.773776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:38.546721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767372+0000","last_change":"2023-03-03T06:18:05.779938+0000","last_active":"2023-03-03T06:18:06.767372+0000","last_peered":"2023-03-03T06:18:06.767372+0000","last_clean":"2023-03-03T06:18:06.767372+0000","last_became_active":"2023-03-03T06:18:05.779669+0000","last_became_peered":"2023-03-03T06:18:05.779669+0000","last_unstale":"2023-03-03T06:18:06.767372+0000","last_undegraded":"2023-03-03T06:18:06.767372+0000","last_fullsized":"2023-03-03T06:18:06.767372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780808+0000","last_change":"2023-03-03T06:18:05.773328+0000","last_active":"2023-03-03T06:18:05.780808+0000","last_peered":"2023-03-03T06:18:05.780808+0000","last_clean":"2023-03-03T06:18:05.780808+0000","last_became_active":"2023-03-03T06:18:05.773065+0000","last_became_peered":"2023-03-03T06:18:05.773065+0000","last_unstale":"2023-03-03T06:18:05.780808+0000","last_undegraded":"2023-03-03T06:18:05.780808+0000","last_fullsized":"2023-03-03T06:18:05.780808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:58.062737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.778686+0000","last_change":"2023-03-03T06:18:05.773425+0000","last_active":"2023-03-03T06:18:05.778686+0000","last_peered":"2023-03-03T06:18:05.778686+0000","last_clean":"2023-03-03T06:18:05.778686+0000","last_became_active":"2023-03-03T06:18:05.772997+0000","last_became_peered":"2023-03-03T06:18:05.772997+0000","last_unstale":"2023-03-03T06:18:05.778686+0000","last_undegraded":"2023-03-03T06:18:05.778686+0000","last_fullsized":"2023-03-03T06:18:05.778686+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:43.144401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773462+0000","last_change":"2023-03-03T06:18:05.771575+0000","last_active":"2023-03-03T06:18:05.773462+0000","last_peered":"2023-03-03T06:18:05.773462+0000","last_clean":"2023-03-03T06:18:05.773462+0000","last_became_active":"2023-03-03T06:18:05.771326+0000","last_became_peered":"2023-03-03T06:18:05.771326+0000","last_unstale":"2023-03-03T06:18:05.773462+0000","last_undegraded":"2023-03-03T06:18:05.773462+0000","last_fullsized":"2023-03-03T06:18:05.773462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781831+0000","last_change":"2023-03-03T06:18:05.773873+0000","last_active":"2023-03-03T06:18:05.781831+0000","last_peered":"2023-03-03T06:18:05.781831+0000","last_clean":"2023-03-03T06:18:05.781831+0000","last_became_active":"2023-03-03T06:18:05.773610+0000","last_became_peered":"2023-03-03T06:18:05.773610+0000","last_unstale":"2023-03-03T06:18:05.781831+0000","last_undegraded":"2023-03-03T06:18:05.781831+0000","last_fullsized":"2023-03-03T06:18:05.781831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:24:21.809616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766274+0000","last_change":"2023-03-03T06:18:05.779587+0000","last_active":"2023-03-03T06:18:06.766274+0000","last_peered":"2023-03-03T06:18:06.766274+0000","last_clean":"2023-03-03T06:18:06.766274+0000","last_became_active":"2023-03-03T06:18:05.779147+0000","last_became_peered":"2023-03-03T06:18:05.779147+0000","last_unstale":"2023-03-03T06:18:06.766274+0000","last_undegraded":"2023-03-03T06:18:06.766274+0000","last_fullsized":"2023-03-03T06:18:06.766274+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766529+0000","last_change":"2023-03-03T06:18:05.774633+0000","last_active":"2023-03-03T06:18:06.766529+0000","last_peered":"2023-03-03T06:18:06.766529+0000","last_clean":"2023-03-03T06:18:06.766529+0000","last_became_active":"2023-03-03T06:18:05.773725+0000","last_became_peered":"2023-03-03T06:18:05.773725+0000","last_unstale":"2023-03-03T06:18:06.766529+0000","last_undegraded":"2023-03-03T06:18:06.766529+0000","last_fullsized":"2023-03-03T06:18:06.766529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.772498+0000","last_change":"2023-03-03T06:18:05.770503+0000","last_active":"2023-03-03T06:18:05.772498+0000","last_peered":"2023-03-03T06:18:05.772498+0000","last_clean":"2023-03-03T06:18:05.772498+0000","last_became_active":"2023-03-03T06:18:05.770093+0000","last_became_peered":"2023-03-03T06:18:05.770093+0000","last_unstale":"2023-03-03T06:18:05.772498+0000","last_undegraded":"2023-03-03T06:18:05.772498+0000","last_fullsized":"2023-03-03T06:18:05.772498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767476+0000","last_change":"2023-03-03T06:18:05.774425+0000","last_active":"2023-03-03T06:18:06.767476+0000","last_peered":"2023-03-03T06:18:06.767476+0000","last_clean":"2023-03-03T06:18:06.767476+0000","last_became_active":"2023-03-03T06:18:05.774137+0000","last_became_peered":"2023-03-03T06:18:05.774137+0000","last_unstale":"2023-03-03T06:18:06.767476+0000","last_undegraded":"2023-03-03T06:18:06.767476+0000","last_fullsized":"2023-03-03T06:18:06.767476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783745+0000","last_change":"2023-03-03T06:18:05.779203+0000","last_active":"2023-03-03T06:18:05.783745+0000","last_peered":"2023-03-03T06:18:05.783745+0000","last_clean":"2023-03-03T06:18:05.783745+0000","last_became_active":"2023-03-03T06:18:05.778887+0000","last_became_peered":"2023-03-03T06:18:05.778887+0000","last_unstale":"2023-03-03T06:18:05.783745+0000","last_undegraded":"2023-03-03T06:18:05.783745+0000","last_fullsized":"2023-03-03T06:18:05.783745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.774328+0000","last_change":"2023-03-03T06:18:05.771290+0000","last_active":"2023-03-03T06:18:05.774328+0000","last_peered":"2023-03-03T06:18:05.774328+0000","last_clean":"2023-03-03T06:18:05.774328+0000","last_became_active":"2023-03-03T06:18:05.771073+0000","last_became_peered":"2023-03-03T06:18:05.771073+0000","last_unstale":"2023-03-03T06:18:05.774328+0000","last_undegraded":"2023-03-03T06:18:05.774328+0000","last_fullsized":"2023-03-03T06:18:05.774328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:01.951025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.774365+0000","last_change":"2023-03-03T06:18:05.772434+0000","last_active":"2023-03-03T06:18:05.774365+0000","last_peered":"2023-03-03T06:18:05.774365+0000","last_clean":"2023-03-03T06:18:05.774365+0000","last_became_active":"2023-03-03T06:18:05.772171+0000","last_became_peered":"2023-03-03T06:18:05.772171+0000","last_unstale":"2023-03-03T06:18:05.774365+0000","last_undegraded":"2023-03-03T06:18:05.774365+0000","last_fullsized":"2023-03-03T06:18:05.774365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:43.200309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766717+0000","last_change":"2023-03-03T06:18:05.775751+0000","last_active":"2023-03-03T06:18:06.766717+0000","last_peered":"2023-03-03T06:18:06.766717+0000","last_clean":"2023-03-03T06:18:06.766717+0000","last_became_active":"2023-03-03T06:18:05.775268+0000","last_became_peered":"2023-03-03T06:18:05.775268+0000","last_unstale":"2023-03-03T06:18:06.766717+0000","last_undegraded":"2023-03-03T06:18:06.766717+0000","last_fullsized":"2023-03-03T06:18:06.766717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766405+0000","last_change":"2023-03-03T06:18:05.779415+0000","last_active":"2023-03-03T06:18:06.766405+0000","last_peered":"2023-03-03T06:18:06.766405+0000","last_clean":"2023-03-03T06:18:06.766405+0000","last_became_active":"2023-03-03T06:18:05.778767+0000","last_became_peered":"2023-03-03T06:18:05.778767+0000","last_unstale":"2023-03-03T06:18:06.766405+0000","last_undegraded":"2023-03-03T06:18:06.766405+0000","last_fullsized":"2023-03-03T06:18:06.766405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.777740+0000","last_change":"2023-03-03T06:18:05.768282+0000","last_active":"2023-03-03T06:18:05.777740+0000","last_peered":"2023-03-03T06:18:05.777740+0000","last_clean":"2023-03-03T06:18:05.777740+0000","last_became_active":"2023-03-03T06:18:05.767972+0000","last_became_peered":"2023-03-03T06:18:05.767972+0000","last_unstale":"2023-03-03T06:18:05.777740+0000","last_undegraded":"2023-03-03T06:18:05.777740+0000","last_fullsized":"2023-03-03T06:18:05.777740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:46.885266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.772479+0000","last_change":"2023-03-03T06:18:05.770349+0000","last_active":"2023-03-03T06:18:05.772479+0000","last_peered":"2023-03-03T06:18:05.772479+0000","last_clean":"2023-03-03T06:18:05.772479+0000","last_became_active":"2023-03-03T06:18:05.769801+0000","last_became_peered":"2023-03-03T06:18:05.769801+0000","last_unstale":"2023-03-03T06:18:05.772479+0000","last_undegraded":"2023-03-03T06:18:05.772479+0000","last_fullsized":"2023-03-03T06:18:05.772479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.777774+0000","last_change":"2023-03-03T06:18:05.769983+0000","last_active":"2023-03-03T06:18:05.777774+0000","last_peered":"2023-03-03T06:18:05.777774+0000","last_clean":"2023-03-03T06:18:05.777774+0000","last_became_active":"2023-03-03T06:18:05.769713+0000","last_became_peered":"2023-03-03T06:18:05.769713+0000","last_unstale":"2023-03-03T06:18:05.777774+0000","last_undegraded":"2023-03-03T06:18:05.777774+0000","last_fullsized":"2023-03-03T06:18:05.777774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:48.612674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.774338+0000","last_change":"2023-03-03T06:18:05.772674+0000","last_active":"2023-03-03T06:18:05.774338+0000","last_peered":"2023-03-03T06:18:05.774338+0000","last_clean":"2023-03-03T06:18:05.774338+0000","last_became_active":"2023-03-03T06:18:05.772420+0000","last_became_peered":"2023-03-03T06:18:05.772420+0000","last_unstale":"2023-03-03T06:18:05.774338+0000","last_undegraded":"2023-03-03T06:18:05.774338+0000","last_fullsized":"2023-03-03T06:18:05.774338+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:17.201682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783869+0000","last_change":"2023-03-03T06:18:05.779465+0000","last_active":"2023-03-03T06:18:05.783869+0000","last_peered":"2023-03-03T06:18:05.783869+0000","last_clean":"2023-03-03T06:18:05.783869+0000","last_became_active":"2023-03-03T06:18:05.779177+0000","last_became_peered":"2023-03-03T06:18:05.779177+0000","last_unstale":"2023-03-03T06:18:05.783869+0000","last_undegraded":"2023-03-03T06:18:05.783869+0000","last_fullsized":"2023-03-03T06:18:05.783869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766718+0000","last_change":"2023-03-03T06:18:05.769339+0000","last_active":"2023-03-03T06:18:06.766718+0000","last_peered":"2023-03-03T06:18:06.766718+0000","last_clean":"2023-03-03T06:18:06.766718+0000","last_became_active":"2023-03-03T06:18:05.768277+0000","last_became_peered":"2023-03-03T06:18:05.768277+0000","last_unstale":"2023-03-03T06:18:06.766718+0000","last_undegraded":"2023-03-03T06:18:06.766718+0000","last_fullsized":"2023-03-03T06:18:06.766718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781531+0000","last_change":"2023-03-03T06:18:05.780097+0000","last_active":"2023-03-03T06:18:05.781531+0000","last_peered":"2023-03-03T06:18:05.781531+0000","last_clean":"2023-03-03T06:18:05.781531+0000","last_became_active":"2023-03-03T06:18:05.779838+0000","last_became_peered":"2023-03-03T06:18:05.779838+0000","last_unstale":"2023-03-03T06:18:05.781531+0000","last_undegraded":"2023-03-03T06:18:05.781531+0000","last_fullsized":"2023-03-03T06:18:05.781531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761222+0000","last_change":"2023-03-03T06:18:05.772869+0000","last_active":"2023-03-03T06:18:06.761222+0000","last_peered":"2023-03-03T06:18:06.761222+0000","last_clean":"2023-03-03T06:18:06.761222+0000","last_became_active":"2023-03-03T06:18:05.772498+0000","last_became_peered":"2023-03-03T06:18:05.772498+0000","last_unstale":"2023-03-03T06:18:06.761222+0000","last_undegraded":"2023-03-03T06:18:06.761222+0000","last_fullsized":"2023-03-03T06:18:06.761222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766506+0000","last_change":"2023-03-03T06:18:05.769326+0000","last_active":"2023-03-03T06:18:06.766506+0000","last_peered":"2023-03-03T06:18:06.766506+0000","last_clean":"2023-03-03T06:18:06.766506+0000","last_became_active":"2023-03-03T06:18:05.768781+0000","last_became_peered":"2023-03-03T06:18:05.768781+0000","last_unstale":"2023-03-03T06:18:06.766506+0000","last_undegraded":"2023-03-03T06:18:06.766506+0000","last_fullsized":"2023-03-03T06:18:06.766506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782707+0000","last_change":"2023-03-03T06:18:05.777824+0000","last_active":"2023-03-03T06:18:05.782707+0000","last_peered":"2023-03-03T06:18:05.782707+0000","last_clean":"2023-03-03T06:18:05.782707+0000","last_became_active":"2023-03-03T06:18:05.777221+0000","last_became_peered":"2023-03-03T06:18:05.777221+0000","last_unstale":"2023-03-03T06:18:05.782707+0000","last_undegraded":"2023-03-03T06:18:05.782707+0000","last_fullsized":"2023-03-03T06:18:05.782707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782704+0000","last_change":"2023-03-03T06:18:05.777434+0000","last_active":"2023-03-03T06:18:05.782704+0000","last_peered":"2023-03-03T06:18:05.782704+0000","last_clean":"2023-03-03T06:18:05.782704+0000","last_became_active":"2023-03-03T06:18:05.776857+0000","last_became_peered":"2023-03-03T06:18:05.776857+0000","last_unstale":"2023-03-03T06:18:05.782704+0000","last_undegraded":"2023-03-03T06:18:05.782704+0000","last_fullsized":"2023-03-03T06:18:05.782704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767237+0000","last_change":"2023-03-03T06:18:05.776802+0000","last_active":"2023-03-03T06:18:06.767237+0000","last_peered":"2023-03-03T06:18:06.767237+0000","last_clean":"2023-03-03T06:18:06.767237+0000","last_became_active":"2023-03-03T06:18:05.776531+0000","last_became_peered":"2023-03-03T06:18:05.776531+0000","last_unstale":"2023-03-03T06:18:06.767237+0000","last_undegraded":"2023-03-03T06:18:06.767237+0000","last_fullsized":"2023-03-03T06:18:06.767237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781661+0000","last_change":"2023-03-03T06:18:05.374916+0000","last_active":"2023-03-03T06:18:05.781661+0000","last_peered":"2023-03-03T06:18:05.781661+0000","last_clean":"2023-03-03T06:18:05.781661+0000","last_became_active":"2023-03-03T06:18:05.374663+0000","last_became_peered":"2023-03-03T06:18:05.374663+0000","last_unstale":"2023-03-03T06:18:05.781661+0000","last_undegraded":"2023-03-03T06:18:05.781661+0000","last_fullsized":"2023-03-03T06:18:05.781661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:19:22.426709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761867+0000","last_change":"2023-03-03T06:18:05.778939+0000","last_active":"2023-03-03T06:18:06.761867+0000","last_peered":"2023-03-03T06:18:06.761867+0000","last_clean":"2023-03-03T06:18:06.761867+0000","last_became_active":"2023-03-03T06:18:05.778350+0000","last_became_peered":"2023-03-03T06:18:05.778350+0000","last_unstale":"2023-03-03T06:18:06.761867+0000","last_undegraded":"2023-03-03T06:18:06.761867+0000","last_fullsized":"2023-03-03T06:18:06.761867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.770605+0000","last_change":"2023-03-03T06:18:05.369187+0000","last_active":"2023-03-03T06:18:05.770605+0000","last_peered":"2023-03-03T06:18:05.770605+0000","last_clean":"2023-03-03T06:18:05.770605+0000","last_became_active":"2023-03-03T06:18:05.368706+0000","last_became_peered":"2023-03-03T06:18:05.368706+0000","last_unstale":"2023-03-03T06:18:05.770605+0000","last_undegraded":"2023-03-03T06:18:05.770605+0000","last_fullsized":"2023-03-03T06:18:05.770605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773787+0000","last_change":"2023-03-03T06:18:05.771810+0000","last_active":"2023-03-03T06:18:05.773787+0000","last_peered":"2023-03-03T06:18:05.773787+0000","last_clean":"2023-03-03T06:18:05.773787+0000","last_became_active":"2023-03-03T06:18:05.771484+0000","last_became_peered":"2023-03-03T06:18:05.771484+0000","last_unstale":"2023-03-03T06:18:05.773787+0000","last_undegraded":"2023-03-03T06:18:05.773787+0000","last_fullsized":"2023-03-03T06:18:05.773787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761169+0000","last_change":"2023-03-03T06:18:05.378838+0000","last_active":"2023-03-03T06:18:06.761169+0000","last_peered":"2023-03-03T06:18:06.761169+0000","last_clean":"2023-03-03T06:18:06.761169+0000","last_became_active":"2023-03-03T06:18:05.378525+0000","last_became_peered":"2023-03-03T06:18:05.378525+0000","last_unstale":"2023-03-03T06:18:06.761169+0000","last_undegraded":"2023-03-03T06:18:06.761169+0000","last_fullsized":"2023-03-03T06:18:06.761169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783276+0000","last_change":"2023-03-03T06:18:05.776315+0000","last_active":"2023-03-03T06:18:05.783276+0000","last_peered":"2023-03-03T06:18:05.783276+0000","last_clean":"2023-03-03T06:18:05.783276+0000","last_became_active":"2023-03-03T06:18:05.775728+0000","last_became_peered":"2023-03-03T06:18:05.775728+0000","last_unstale":"2023-03-03T06:18:05.783276+0000","last_undegraded":"2023-03-03T06:18:05.783276+0000","last_fullsized":"2023-03-03T06:18:05.783276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:17.287057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.763700+0000","last_change":"2023-03-03T06:18:05.378086+0000","last_active":"2023-03-03T06:18:05.763700+0000","last_peered":"2023-03-03T06:18:05.763700+0000","last_clean":"2023-03-03T06:18:05.763700+0000","last_became_active":"2023-03-03T06:18:05.377265+0000","last_became_peered":"2023-03-03T06:18:05.377265+0000","last_unstale":"2023-03-03T06:18:05.763700+0000","last_undegraded":"2023-03-03T06:18:05.763700+0000","last_fullsized":"2023-03-03T06:18:05.763700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761128+0000","last_change":"2023-03-03T06:18:05.778565+0000","last_active":"2023-03-03T06:18:06.761128+0000","last_peered":"2023-03-03T06:18:06.761128+0000","last_clean":"2023-03-03T06:18:06.761128+0000","last_became_active":"2023-03-03T06:18:05.778231+0000","last_became_peered":"2023-03-03T06:18:05.778231+0000","last_unstale":"2023-03-03T06:18:06.761128+0000","last_undegraded":"2023-03-03T06:18:06.761128+0000","last_fullsized":"2023-03-03T06:18:06.761128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767346+0000","last_change":"2023-03-03T06:18:05.370256+0000","last_active":"2023-03-03T06:18:06.767346+0000","last_peered":"2023-03-03T06:18:06.767346+0000","last_clean":"2023-03-03T06:18:06.767346+0000","last_became_active":"2023-03-03T06:18:05.369970+0000","last_became_peered":"2023-03-03T06:18:05.369970+0000","last_unstale":"2023-03-03T06:18:06.767346+0000","last_undegraded":"2023-03-03T06:18:06.767346+0000","last_fullsized":"2023-03-03T06:18:06.767346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761149+0000","last_change":"2023-03-03T06:18:05.766019+0000","last_active":"2023-03-03T06:18:06.761149+0000","last_peered":"2023-03-03T06:18:06.761149+0000","last_clean":"2023-03-03T06:18:06.761149+0000","last_became_active":"2023-03-03T06:18:05.765768+0000","last_became_peered":"2023-03-03T06:18:05.765768+0000","last_unstale":"2023-03-03T06:18:06.761149+0000","last_undegraded":"2023-03-03T06:18:06.761149+0000","last_fullsized":"2023-03-03T06:18:06.761149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766991+0000","last_change":"2023-03-03T06:18:05.367491+0000","last_active":"2023-03-03T06:18:06.766991+0000","last_peered":"2023-03-03T06:18:06.766991+0000","last_clean":"2023-03-03T06:18:06.766991+0000","last_became_active":"2023-03-03T06:18:05.367177+0000","last_became_peered":"2023-03-03T06:18:05.367177+0000","last_unstale":"2023-03-03T06:18:06.766991+0000","last_undegraded":"2023-03-03T06:18:06.766991+0000","last_fullsized":"2023-03-03T06:18:06.766991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:55.267658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782754+0000","last_change":"2023-03-03T06:18:05.778585+0000","last_active":"2023-03-03T06:18:05.782754+0000","last_peered":"2023-03-03T06:18:05.782754+0000","last_clean":"2023-03-03T06:18:05.782754+0000","last_became_active":"2023-03-03T06:18:05.777053+0000","last_became_peered":"2023-03-03T06:18:05.777053+0000","last_unstale":"2023-03-03T06:18:05.782754+0000","last_undegraded":"2023-03-03T06:18:05.782754+0000","last_fullsized":"2023-03-03T06:18:05.782754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.786362+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:05.786362+0000","last_peered":"2023-03-03T06:18:05.786362+0000","last_clean":"2023-03-03T06:18:05.786362+0000","last_became_active":"2023-03-03T06:18:05.372051+0000","last_became_peered":"2023-03-03T06:18:05.372051+0000","last_unstale":"2023-03-03T06:18:05.786362+0000","last_undegraded":"2023-03-03T06:18:05.786362+0000","last_fullsized":"2023-03-03T06:18:05.786362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783758+0000","last_change":"2023-03-03T06:18:05.779300+0000","last_active":"2023-03-03T06:18:05.783758+0000","last_peered":"2023-03-03T06:18:05.783758+0000","last_clean":"2023-03-03T06:18:05.783758+0000","last_became_active":"2023-03-03T06:18:05.778878+0000","last_became_peered":"2023-03-03T06:18:05.778878+0000","last_unstale":"2023-03-03T06:18:05.783758+0000","last_undegraded":"2023-03-03T06:18:05.783758+0000","last_fullsized":"2023-03-03T06:18:05.783758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766613+0000","last_change":"2023-03-03T06:18:05.377498+0000","last_active":"2023-03-03T06:18:06.766613+0000","last_peered":"2023-03-03T06:18:06.766613+0000","last_clean":"2023-03-03T06:18:06.766613+0000","last_became_active":"2023-03-03T06:18:05.377168+0000","last_became_peered":"2023-03-03T06:18:05.377168+0000","last_unstale":"2023-03-03T06:18:06.766613+0000","last_undegraded":"2023-03-03T06:18:06.766613+0000","last_fullsized":"2023-03-03T06:18:06.766613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:38.382151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761390+0000","last_change":"2023-03-03T06:18:05.779094+0000","last_active":"2023-03-03T06:18:06.761390+0000","last_peered":"2023-03-03T06:18:06.761390+0000","last_clean":"2023-03-03T06:18:06.761390+0000","last_became_active":"2023-03-03T06:18:05.778802+0000","last_became_peered":"2023-03-03T06:18:05.778802+0000","last_unstale":"2023-03-03T06:18:06.761390+0000","last_undegraded":"2023-03-03T06:18:06.761390+0000","last_fullsized":"2023-03-03T06:18:06.761390+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.759335+0000","last_change":"2023-03-03T06:18:05.376181+0000","last_active":"2023-03-03T06:18:05.759335+0000","last_peered":"2023-03-03T06:18:05.759335+0000","last_clean":"2023-03-03T06:18:05.759335+0000","last_became_active":"2023-03-03T06:18:05.375857+0000","last_became_peered":"2023-03-03T06:18:05.375857+0000","last_unstale":"2023-03-03T06:18:05.759335+0000","last_undegraded":"2023-03-03T06:18:05.759335+0000","last_fullsized":"2023-03-03T06:18:05.759335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767254+0000","last_change":"2023-03-03T06:18:05.775649+0000","last_active":"2023-03-03T06:18:06.767254+0000","last_peered":"2023-03-03T06:18:06.767254+0000","last_clean":"2023-03-03T06:18:06.767254+0000","last_became_active":"2023-03-03T06:18:05.775370+0000","last_became_peered":"2023-03-03T06:18:05.775370+0000","last_unstale":"2023-03-03T06:18:06.767254+0000","last_undegraded":"2023-03-03T06:18:06.767254+0000","last_fullsized":"2023-03-03T06:18:06.767254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766285+0000","last_change":"2023-03-03T06:18:05.377478+0000","last_active":"2023-03-03T06:18:06.766285+0000","last_peered":"2023-03-03T06:18:06.766285+0000","last_clean":"2023-03-03T06:18:06.766285+0000","last_became_active":"2023-03-03T06:18:05.377177+0000","last_became_peered":"2023-03-03T06:18:05.377177+0000","last_unstale":"2023-03-03T06:18:06.766285+0000","last_undegraded":"2023-03-03T06:18:06.766285+0000","last_fullsized":"2023-03-03T06:18:06.766285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:42.138865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.773807+0000","last_change":"2023-03-03T06:18:05.771983+0000","last_active":"2023-03-03T06:18:05.773807+0000","last_peered":"2023-03-03T06:18:05.773807+0000","last_clean":"2023-03-03T06:18:05.773807+0000","last_became_active":"2023-03-03T06:18:05.771526+0000","last_became_peered":"2023-03-03T06:18:05.771526+0000","last_unstale":"2023-03-03T06:18:05.773807+0000","last_undegraded":"2023-03-03T06:18:05.773807+0000","last_fullsized":"2023-03-03T06:18:05.773807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783605+0000","last_change":"2023-03-03T06:18:05.379123+0000","last_active":"2023-03-03T06:18:05.783605+0000","last_peered":"2023-03-03T06:18:05.783605+0000","last_clean":"2023-03-03T06:18:05.783605+0000","last_became_active":"2023-03-03T06:18:05.378826+0000","last_became_peered":"2023-03-03T06:18:05.378826+0000","last_unstale":"2023-03-03T06:18:05.783605+0000","last_undegraded":"2023-03-03T06:18:05.783605+0000","last_fullsized":"2023-03-03T06:18:05.783605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761645+0000","last_change":"2023-03-03T06:18:05.766998+0000","last_active":"2023-03-03T06:18:06.761645+0000","last_peered":"2023-03-03T06:18:06.761645+0000","last_clean":"2023-03-03T06:18:06.761645+0000","last_became_active":"2023-03-03T06:18:05.764785+0000","last_became_peered":"2023-03-03T06:18:05.764785+0000","last_unstale":"2023-03-03T06:18:06.761645+0000","last_undegraded":"2023-03-03T06:18:06.761645+0000","last_fullsized":"2023-03-03T06:18:06.761645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767168+0000","last_change":"2023-03-03T06:18:05.377431+0000","last_active":"2023-03-03T06:18:06.767168+0000","last_peered":"2023-03-03T06:18:06.767168+0000","last_clean":"2023-03-03T06:18:06.767168+0000","last_became_active":"2023-03-03T06:18:05.377107+0000","last_became_peered":"2023-03-03T06:18:05.377107+0000","last_unstale":"2023-03-03T06:18:06.767168+0000","last_undegraded":"2023-03-03T06:18:06.767168+0000","last_fullsized":"2023-03-03T06:18:06.767168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766647+0000","last_change":"2023-03-03T06:18:05.774893+0000","last_active":"2023-03-03T06:18:06.766647+0000","last_peered":"2023-03-03T06:18:06.766647+0000","last_clean":"2023-03-03T06:18:06.766647+0000","last_became_active":"2023-03-03T06:18:05.774256+0000","last_became_peered":"2023-03-03T06:18:05.774256+0000","last_unstale":"2023-03-03T06:18:06.766647+0000","last_undegraded":"2023-03-03T06:18:06.766647+0000","last_fullsized":"2023-03-03T06:18:06.766647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766744+0000","last_change":"2023-03-03T06:18:05.377497+0000","last_active":"2023-03-03T06:18:06.766744+0000","last_peered":"2023-03-03T06:18:06.766744+0000","last_clean":"2023-03-03T06:18:06.766744+0000","last_became_active":"2023-03-03T06:18:05.377215+0000","last_became_peered":"2023-03-03T06:18:05.377215+0000","last_unstale":"2023-03-03T06:18:06.766744+0000","last_undegraded":"2023-03-03T06:18:06.766744+0000","last_fullsized":"2023-03-03T06:18:06.766744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783973+0000","last_change":"2023-03-03T06:18:05.781538+0000","last_active":"2023-03-03T06:18:05.783973+0000","last_peered":"2023-03-03T06:18:05.783973+0000","last_clean":"2023-03-03T06:18:05.783973+0000","last_became_active":"2023-03-03T06:18:05.781285+0000","last_became_peered":"2023-03-03T06:18:05.781285+0000","last_unstale":"2023-03-03T06:18:05.783973+0000","last_undegraded":"2023-03-03T06:18:05.783973+0000","last_fullsized":"2023-03-03T06:18:05.783973+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767086+0000","last_change":"2023-03-03T06:18:05.369584+0000","last_active":"2023-03-03T06:18:06.767086+0000","last_peered":"2023-03-03T06:18:06.767086+0000","last_clean":"2023-03-03T06:18:06.767086+0000","last_became_active":"2023-03-03T06:18:05.368904+0000","last_became_peered":"2023-03-03T06:18:05.368904+0000","last_unstale":"2023-03-03T06:18:06.767086+0000","last_undegraded":"2023-03-03T06:18:06.767086+0000","last_fullsized":"2023-03-03T06:18:06.767086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:23.861622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.786136+0000","last_change":"2023-03-03T06:18:05.779823+0000","last_active":"2023-03-03T06:18:05.786136+0000","last_peered":"2023-03-03T06:18:05.786136+0000","last_clean":"2023-03-03T06:18:05.786136+0000","last_became_active":"2023-03-03T06:18:05.779561+0000","last_became_peered":"2023-03-03T06:18:05.779561+0000","last_unstale":"2023-03-03T06:18:05.786136+0000","last_undegraded":"2023-03-03T06:18:05.786136+0000","last_fullsized":"2023-03-03T06:18:05.786136+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:54.968231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767196+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:06.767196+0000","last_peered":"2023-03-03T06:18:06.767196+0000","last_clean":"2023-03-03T06:18:06.767196+0000","last_became_active":"2023-03-03T06:18:05.372057+0000","last_became_peered":"2023-03-03T06:18:05.372057+0000","last_unstale":"2023-03-03T06:18:06.767196+0000","last_undegraded":"2023-03-03T06:18:06.767196+0000","last_fullsized":"2023-03-03T06:18:06.767196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782200+0000","last_change":"2023-03-03T06:18:05.780035+0000","last_active":"2023-03-03T06:18:05.782200+0000","last_peered":"2023-03-03T06:18:05.782200+0000","last_clean":"2023-03-03T06:18:05.782200+0000","last_became_active":"2023-03-03T06:18:05.779650+0000","last_became_peered":"2023-03-03T06:18:05.779650+0000","last_unstale":"2023-03-03T06:18:05.782200+0000","last_undegraded":"2023-03-03T06:18:05.782200+0000","last_fullsized":"2023-03-03T06:18:05.782200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761617+0000","last_change":"2023-03-03T06:18:05.379968+0000","last_active":"2023-03-03T06:18:06.761617+0000","last_peered":"2023-03-03T06:18:06.761617+0000","last_clean":"2023-03-03T06:18:06.761617+0000","last_became_active":"2023-03-03T06:18:05.379635+0000","last_became_peered":"2023-03-03T06:18:05.379635+0000","last_unstale":"2023-03-03T06:18:06.761617+0000","last_undegraded":"2023-03-03T06:18:06.761617+0000","last_fullsized":"2023-03-03T06:18:06.761617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.777783+0000","last_change":"2023-03-03T06:18:05.768972+0000","last_active":"2023-03-03T06:18:05.777783+0000","last_peered":"2023-03-03T06:18:05.777783+0000","last_clean":"2023-03-03T06:18:05.777783+0000","last_became_active":"2023-03-03T06:18:05.768647+0000","last_became_peered":"2023-03-03T06:18:05.768647+0000","last_unstale":"2023-03-03T06:18:05.777783+0000","last_undegraded":"2023-03-03T06:18:05.777783+0000","last_fullsized":"2023-03-03T06:18:05.777783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:15:35.033362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.765000+0000","last_change":"2023-03-03T06:18:05.378643+0000","last_active":"2023-03-03T06:18:05.765000+0000","last_peered":"2023-03-03T06:18:05.765000+0000","last_clean":"2023-03-03T06:18:05.765000+0000","last_became_active":"2023-03-03T06:18:05.378333+0000","last_became_peered":"2023-03-03T06:18:05.378333+0000","last_unstale":"2023-03-03T06:18:05.765000+0000","last_undegraded":"2023-03-03T06:18:05.765000+0000","last_fullsized":"2023-03-03T06:18:05.765000+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780860+0000","last_change":"2023-03-03T06:18:05.774576+0000","last_active":"2023-03-03T06:18:05.780860+0000","last_peered":"2023-03-03T06:18:05.780860+0000","last_clean":"2023-03-03T06:18:05.780860+0000","last_became_active":"2023-03-03T06:18:05.774278+0000","last_became_peered":"2023-03-03T06:18:05.774278+0000","last_unstale":"2023-03-03T06:18:05.780860+0000","last_undegraded":"2023-03-03T06:18:05.780860+0000","last_fullsized":"2023-03-03T06:18:05.780860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:20.212207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.767475+0000","last_change":"2023-03-03T06:18:05.374483+0000","last_active":"2023-03-03T06:18:05.767475+0000","last_peered":"2023-03-03T06:18:05.767475+0000","last_clean":"2023-03-03T06:18:05.767475+0000","last_became_active":"2023-03-03T06:18:05.374224+0000","last_became_peered":"2023-03-03T06:18:05.374224+0000","last_unstale":"2023-03-03T06:18:05.767475+0000","last_undegraded":"2023-03-03T06:18:05.767475+0000","last_fullsized":"2023-03-03T06:18:05.767475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783938+0000","last_change":"2023-03-03T06:18:05.779652+0000","last_active":"2023-03-03T06:18:05.783938+0000","last_peered":"2023-03-03T06:18:05.783938+0000","last_clean":"2023-03-03T06:18:05.783938+0000","last_became_active":"2023-03-03T06:18:05.778557+0000","last_became_peered":"2023-03-03T06:18:05.778557+0000","last_unstale":"2023-03-03T06:18:05.783938+0000","last_undegraded":"2023-03-03T06:18:05.783938+0000","last_fullsized":"2023-03-03T06:18:05.783938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761087+0000","last_change":"2023-03-03T06:18:05.373177+0000","last_active":"2023-03-03T06:18:06.761087+0000","last_peered":"2023-03-03T06:18:06.761087+0000","last_clean":"2023-03-03T06:18:06.761087+0000","last_became_active":"2023-03-03T06:18:05.372879+0000","last_became_peered":"2023-03-03T06:18:05.372879+0000","last_unstale":"2023-03-03T06:18:06.761087+0000","last_undegraded":"2023-03-03T06:18:06.761087+0000","last_fullsized":"2023-03-03T06:18:06.761087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782728+0000","last_change":"2023-03-03T06:18:05.778960+0000","last_active":"2023-03-03T06:18:05.782728+0000","last_peered":"2023-03-03T06:18:05.782728+0000","last_clean":"2023-03-03T06:18:05.782728+0000","last_became_active":"2023-03-03T06:18:05.777570+0000","last_became_peered":"2023-03-03T06:18:05.777570+0000","last_unstale":"2023-03-03T06:18:05.782728+0000","last_undegraded":"2023-03-03T06:18:05.782728+0000","last_fullsized":"2023-03-03T06:18:05.782728+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.357551+0000","last_change":"2023-03-03T06:18:00.701360+0000","last_active":"2023-03-03T06:18:05.357551+0000","last_peered":"2023-03-03T06:18:05.357551+0000","last_clean":"2023-03-03T06:18:05.357551+0000","last_became_active":"2023-03-03T06:18:00.701042+0000","last_became_peered":"2023-03-03T06:18:00.701042+0000","last_unstale":"2023-03-03T06:18:05.357551+0000","last_undegraded":"2023-03-03T06:18:05.357551+0000","last_fullsized":"2023-03-03T06:18:05.357551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:10.253889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767335+0000","last_change":"2023-03-03T06:18:05.372375+0000","last_active":"2023-03-03T06:18:06.767335+0000","last_peered":"2023-03-03T06:18:06.767335+0000","last_clean":"2023-03-03T06:18:06.767335+0000","last_became_active":"2023-03-03T06:18:05.371680+0000","last_became_peered":"2023-03-03T06:18:05.371680+0000","last_unstale":"2023-03-03T06:18:06.767335+0000","last_undegraded":"2023-03-03T06:18:06.767335+0000","last_fullsized":"2023-03-03T06:18:06.767335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782775+0000","last_change":"2023-03-03T06:18:05.778931+0000","last_active":"2023-03-03T06:18:05.782775+0000","last_peered":"2023-03-03T06:18:05.782775+0000","last_clean":"2023-03-03T06:18:05.782775+0000","last_became_active":"2023-03-03T06:18:05.778723+0000","last_became_peered":"2023-03-03T06:18:05.778723+0000","last_unstale":"2023-03-03T06:18:05.782775+0000","last_undegraded":"2023-03-03T06:18:05.782775+0000","last_fullsized":"2023-03-03T06:18:05.782775+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.764427+0000","last_change":"2023-03-03T06:18:00.351410+0000","last_active":"2023-03-03T06:18:05.764427+0000","last_peered":"2023-03-03T06:18:05.764427+0000","last_clean":"2023-03-03T06:18:05.764427+0000","last_became_active":"2023-03-03T06:18:00.351036+0000","last_became_peered":"2023-03-03T06:18:00.351036+0000","last_unstale":"2023-03-03T06:18:05.764427+0000","last_undegraded":"2023-03-03T06:18:05.764427+0000","last_fullsized":"2023-03-03T06:18:05.764427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.763797+0000","last_change":"2023-03-03T06:18:05.378884+0000","last_active":"2023-03-03T06:18:05.763797+0000","last_peered":"2023-03-03T06:18:05.763797+0000","last_clean":"2023-03-03T06:18:05.763797+0000","last_became_active":"2023-03-03T06:18:05.378597+0000","last_became_peered":"2023-03-03T06:18:05.378597+0000","last_unstale":"2023-03-03T06:18:05.763797+0000","last_undegraded":"2023-03-03T06:18:05.763797+0000","last_fullsized":"2023-03-03T06:18:05.763797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780959+0000","last_change":"2023-03-03T06:18:05.777398+0000","last_active":"2023-03-03T06:18:05.780959+0000","last_peered":"2023-03-03T06:18:05.780959+0000","last_clean":"2023-03-03T06:18:05.780959+0000","last_became_active":"2023-03-03T06:18:05.777065+0000","last_became_peered":"2023-03-03T06:18:05.777065+0000","last_unstale":"2023-03-03T06:18:05.780959+0000","last_undegraded":"2023-03-03T06:18:05.780959+0000","last_fullsized":"2023-03-03T06:18:05.780959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.365191+0000","last_change":"2023-03-03T06:18:00.351233+0000","last_active":"2023-03-03T06:18:05.365191+0000","last_peered":"2023-03-03T06:18:05.365191+0000","last_clean":"2023-03-03T06:18:05.365191+0000","last_became_active":"2023-03-03T06:18:00.350944+0000","last_became_peered":"2023-03-03T06:18:00.350944+0000","last_unstale":"2023-03-03T06:18:05.365191+0000","last_undegraded":"2023-03-03T06:18:05.365191+0000","last_fullsized":"2023-03-03T06:18:05.365191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:08:23.569811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.770017+0000","last_change":"2023-03-03T06:18:05.374876+0000","last_active":"2023-03-03T06:18:05.770017+0000","last_peered":"2023-03-03T06:18:05.770017+0000","last_clean":"2023-03-03T06:18:05.770017+0000","last_became_active":"2023-03-03T06:18:05.374266+0000","last_became_peered":"2023-03-03T06:18:05.374266+0000","last_unstale":"2023-03-03T06:18:05.770017+0000","last_undegraded":"2023-03-03T06:18:05.770017+0000","last_fullsized":"2023-03-03T06:18:05.770017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.782760+0000","last_change":"2023-03-03T06:18:05.779139+0000","last_active":"2023-03-03T06:18:05.782760+0000","last_peered":"2023-03-03T06:18:05.782760+0000","last_clean":"2023-03-03T06:18:05.782760+0000","last_became_active":"2023-03-03T06:18:05.777315+0000","last_became_peered":"2023-03-03T06:18:05.777315+0000","last_unstale":"2023-03-03T06:18:05.782760+0000","last_undegraded":"2023-03-03T06:18:05.782760+0000","last_fullsized":"2023-03-03T06:18:05.782760+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.762243+0000","last_change":"2023-03-03T06:18:00.351257+0000","last_active":"2023-03-03T06:18:05.762243+0000","last_peered":"2023-03-03T06:18:05.762243+0000","last_clean":"2023-03-03T06:18:05.762243+0000","last_became_active":"2023-03-03T06:18:00.350933+0000","last_became_peered":"2023-03-03T06:18:00.350933+0000","last_unstale":"2023-03-03T06:18:05.762243+0000","last_undegraded":"2023-03-03T06:18:05.762243+0000","last_fullsized":"2023-03-03T06:18:05.762243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758385+0000","last_change":"2023-03-03T06:17:54.307666+0000","last_active":"2023-03-03T06:18:05.758385+0000","last_peered":"2023-03-03T06:18:05.758385+0000","last_clean":"2023-03-03T06:18:05.758385+0000","last_became_active":"2023-03-03T06:17:54.306552+0000","last_became_peered":"2023-03-03T06:17:54.306552+0000","last_unstale":"2023-03-03T06:18:05.758385+0000","last_undegraded":"2023-03-03T06:18:05.758385+0000","last_fullsized":"2023-03-03T06:18:05.758385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_clean_scrub_stamp":"2023-03-03T06:17:53.091715+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:56:41.267904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761267+0000","last_change":"2023-03-03T06:18:05.366688+0000","last_active":"2023-03-03T06:18:06.761267+0000","last_peered":"2023-03-03T06:18:06.761267+0000","last_clean":"2023-03-03T06:18:06.761267+0000","last_became_active":"2023-03-03T06:18:05.366412+0000","last_became_peered":"2023-03-03T06:18:05.366412+0000","last_unstale":"2023-03-03T06:18:06.761267+0000","last_undegraded":"2023-03-03T06:18:06.761267+0000","last_fullsized":"2023-03-03T06:18:06.761267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.786219+0000","last_change":"2023-03-03T06:18:05.781486+0000","last_active":"2023-03-03T06:18:05.786219+0000","last_peered":"2023-03-03T06:18:05.786219+0000","last_clean":"2023-03-03T06:18:05.786219+0000","last_became_active":"2023-03-03T06:18:05.781220+0000","last_became_peered":"2023-03-03T06:18:05.781220+0000","last_unstale":"2023-03-03T06:18:05.786219+0000","last_undegraded":"2023-03-03T06:18:05.786219+0000","last_fullsized":"2023-03-03T06:18:05.786219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:39.387687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758932+0000","last_change":"2023-03-03T06:18:00.345967+0000","last_active":"2023-03-03T06:18:05.758932+0000","last_peered":"2023-03-03T06:18:05.758932+0000","last_clean":"2023-03-03T06:18:05.758932+0000","last_became_active":"2023-03-03T06:18:00.345666+0000","last_became_peered":"2023-03-03T06:18:00.345666+0000","last_unstale":"2023-03-03T06:18:05.758932+0000","last_undegraded":"2023-03-03T06:18:05.758932+0000","last_fullsized":"2023-03-03T06:18:05.758932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.760333+0000","last_change":"2023-03-03T06:18:05.379214+0000","last_active":"2023-03-03T06:18:05.760333+0000","last_peered":"2023-03-03T06:18:05.760333+0000","last_clean":"2023-03-03T06:18:05.760333+0000","last_became_active":"2023-03-03T06:18:05.378867+0000","last_became_peered":"2023-03-03T06:18:05.378867+0000","last_unstale":"2023-03-03T06:18:05.760333+0000","last_undegraded":"2023-03-03T06:18:05.760333+0000","last_fullsized":"2023-03-03T06:18:05.760333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783637+0000","last_change":"2023-03-03T06:18:05.782275+0000","last_active":"2023-03-03T06:18:05.783637+0000","last_peered":"2023-03-03T06:18:05.783637+0000","last_clean":"2023-03-03T06:18:05.783637+0000","last_became_active":"2023-03-03T06:18:05.781968+0000","last_became_peered":"2023-03-03T06:18:05.781968+0000","last_unstale":"2023-03-03T06:18:05.783637+0000","last_undegraded":"2023-03-03T06:18:05.783637+0000","last_fullsized":"2023-03-03T06:18:05.783637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.759159+0000","last_change":"2023-03-03T06:18:00.701652+0000","last_active":"2023-03-03T06:18:05.759159+0000","last_peered":"2023-03-03T06:18:05.759159+0000","last_clean":"2023-03-03T06:18:05.759159+0000","last_became_active":"2023-03-03T06:18:00.701382+0000","last_became_peered":"2023-03-03T06:18:00.701382+0000","last_unstale":"2023-03-03T06:18:05.759159+0000","last_undegraded":"2023-03-03T06:18:05.759159+0000","last_fullsized":"2023-03-03T06:18:05.759159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:50:16.060640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.775494+0000","last_change":"2023-03-03T06:18:05.374785+0000","last_active":"2023-03-03T06:18:05.775494+0000","last_peered":"2023-03-03T06:18:05.775494+0000","last_clean":"2023-03-03T06:18:05.775494+0000","last_became_active":"2023-03-03T06:18:05.374322+0000","last_became_peered":"2023-03-03T06:18:05.374322+0000","last_unstale":"2023-03-03T06:18:05.775494+0000","last_undegraded":"2023-03-03T06:18:05.775494+0000","last_fullsized":"2023-03-03T06:18:05.775494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.779863+0000","last_change":"2023-03-03T06:18:05.775603+0000","last_active":"2023-03-03T06:18:05.779863+0000","last_peered":"2023-03-03T06:18:05.779863+0000","last_clean":"2023-03-03T06:18:05.779863+0000","last_became_active":"2023-03-03T06:18:05.775350+0000","last_became_peered":"2023-03-03T06:18:05.775350+0000","last_unstale":"2023-03-03T06:18:05.779863+0000","last_undegraded":"2023-03-03T06:18:05.779863+0000","last_fullsized":"2023-03-03T06:18:05.779863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761609+0000","last_change":"2023-03-03T06:18:05.371388+0000","last_active":"2023-03-03T06:18:06.761609+0000","last_peered":"2023-03-03T06:18:06.761609+0000","last_clean":"2023-03-03T06:18:06.761609+0000","last_became_active":"2023-03-03T06:18:05.371026+0000","last_became_peered":"2023-03-03T06:18:05.371026+0000","last_unstale":"2023-03-03T06:18:06.761609+0000","last_undegraded":"2023-03-03T06:18:06.761609+0000","last_fullsized":"2023-03-03T06:18:06.761609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.357510+0000","last_change":"2023-03-03T06:18:00.349337+0000","last_active":"2023-03-03T06:18:05.357510+0000","last_peered":"2023-03-03T06:18:05.357510+0000","last_clean":"2023-03-03T06:18:05.357510+0000","last_became_active":"2023-03-03T06:18:00.349101+0000","last_became_peered":"2023-03-03T06:18:00.349101+0000","last_unstale":"2023-03-03T06:18:05.357510+0000","last_undegraded":"2023-03-03T06:18:05.357510+0000","last_fullsized":"2023-03-03T06:18:05.357510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761022+0000","last_change":"2023-03-03T06:18:05.762420+0000","last_active":"2023-03-03T06:18:06.761022+0000","last_peered":"2023-03-03T06:18:06.761022+0000","last_clean":"2023-03-03T06:18:06.761022+0000","last_became_active":"2023-03-03T06:18:05.760911+0000","last_became_peered":"2023-03-03T06:18:05.760911+0000","last_unstale":"2023-03-03T06:18:06.761022+0000","last_undegraded":"2023-03-03T06:18:06.761022+0000","last_fullsized":"2023-03-03T06:18:06.761022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.365729+0000","last_change":"2023-03-03T06:18:00.352230+0000","last_active":"2023-03-03T06:18:05.365729+0000","last_peered":"2023-03-03T06:18:05.365729+0000","last_clean":"2023-03-03T06:18:05.365729+0000","last_became_active":"2023-03-03T06:18:00.351881+0000","last_became_peered":"2023-03-03T06:18:00.351881+0000","last_unstale":"2023-03-03T06:18:05.365729+0000","last_undegraded":"2023-03-03T06:18:05.365729+0000","last_fullsized":"2023-03-03T06:18:05.365729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:18:24.984007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.760860+0000","last_change":"2023-03-03T06:18:05.382451+0000","last_active":"2023-03-03T06:18:05.760860+0000","last_peered":"2023-03-03T06:18:05.760860+0000","last_clean":"2023-03-03T06:18:05.760860+0000","last_became_active":"2023-03-03T06:18:05.382112+0000","last_became_peered":"2023-03-03T06:18:05.382112+0000","last_unstale":"2023-03-03T06:18:05.760860+0000","last_undegraded":"2023-03-03T06:18:05.760860+0000","last_fullsized":"2023-03-03T06:18:05.760860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783537+0000","last_change":"2023-03-03T06:18:05.777771+0000","last_active":"2023-03-03T06:18:05.783537+0000","last_peered":"2023-03-03T06:18:05.783537+0000","last_clean":"2023-03-03T06:18:05.783537+0000","last_became_active":"2023-03-03T06:18:05.777457+0000","last_became_peered":"2023-03-03T06:18:05.777457+0000","last_unstale":"2023-03-03T06:18:05.783537+0000","last_undegraded":"2023-03-03T06:18:05.783537+0000","last_fullsized":"2023-03-03T06:18:05.783537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:24.867158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.771241+0000","last_change":"2023-03-03T06:18:05.382340+0000","last_active":"2023-03-03T06:18:05.771241+0000","last_peered":"2023-03-03T06:18:05.771241+0000","last_clean":"2023-03-03T06:18:05.771241+0000","last_became_active":"2023-03-03T06:18:05.382051+0000","last_became_peered":"2023-03-03T06:18:05.382051+0000","last_unstale":"2023-03-03T06:18:05.771241+0000","last_undegraded":"2023-03-03T06:18:05.771241+0000","last_fullsized":"2023-03-03T06:18:05.771241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783095+0000","last_change":"2023-03-03T06:18:05.778551+0000","last_active":"2023-03-03T06:18:05.783095+0000","last_peered":"2023-03-03T06:18:05.783095+0000","last_clean":"2023-03-03T06:18:05.783095+0000","last_became_active":"2023-03-03T06:18:05.778166+0000","last_became_peered":"2023-03-03T06:18:05.778166+0000","last_unstale":"2023-03-03T06:18:05.783095+0000","last_undegraded":"2023-03-03T06:18:05.783095+0000","last_fullsized":"2023-03-03T06:18:05.783095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:39.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766897+0000","last_change":"2023-03-03T06:18:05.371014+0000","last_active":"2023-03-03T06:18:06.766897+0000","last_peered":"2023-03-03T06:18:06.766897+0000","last_clean":"2023-03-03T06:18:06.766897+0000","last_became_active":"2023-03-03T06:18:05.370730+0000","last_became_peered":"2023-03-03T06:18:05.370730+0000","last_unstale":"2023-03-03T06:18:06.766897+0000","last_undegraded":"2023-03-03T06:18:06.766897+0000","last_fullsized":"2023-03-03T06:18:06.766897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780783+0000","last_change":"2023-03-03T06:18:05.772599+0000","last_active":"2023-03-03T06:18:05.780783+0000","last_peered":"2023-03-03T06:18:05.780783+0000","last_clean":"2023-03-03T06:18:05.780783+0000","last_became_active":"2023-03-03T06:18:05.772305+0000","last_became_peered":"2023-03-03T06:18:05.772305+0000","last_unstale":"2023-03-03T06:18:05.780783+0000","last_undegraded":"2023-03-03T06:18:05.780783+0000","last_fullsized":"2023-03-03T06:18:05.780783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:14.254976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767411+0000","last_change":"2023-03-03T06:18:05.374466+0000","last_active":"2023-03-03T06:18:06.767411+0000","last_peered":"2023-03-03T06:18:06.767411+0000","last_clean":"2023-03-03T06:18:06.767411+0000","last_became_active":"2023-03-03T06:18:05.374194+0000","last_became_peered":"2023-03-03T06:18:05.374194+0000","last_unstale":"2023-03-03T06:18:06.767411+0000","last_undegraded":"2023-03-03T06:18:06.767411+0000","last_fullsized":"2023-03-03T06:18:06.767411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.781812+0000","last_change":"2023-03-03T06:18:05.773340+0000","last_active":"2023-03-03T06:18:05.781812+0000","last_peered":"2023-03-03T06:18:05.781812+0000","last_clean":"2023-03-03T06:18:05.781812+0000","last_became_active":"2023-03-03T06:18:05.772391+0000","last_became_peered":"2023-03-03T06:18:05.772391+0000","last_unstale":"2023-03-03T06:18:05.781812+0000","last_undegraded":"2023-03-03T06:18:05.781812+0000","last_fullsized":"2023-03-03T06:18:05.781812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.762090+0000","last_change":"2023-03-03T06:18:05.369911+0000","last_active":"2023-03-03T06:18:06.762090+0000","last_peered":"2023-03-03T06:18:06.762090+0000","last_clean":"2023-03-03T06:18:06.762090+0000","last_became_active":"2023-03-03T06:18:05.369605+0000","last_became_peered":"2023-03-03T06:18:05.369605+0000","last_unstale":"2023-03-03T06:18:06.762090+0000","last_undegraded":"2023-03-03T06:18:06.762090+0000","last_fullsized":"2023-03-03T06:18:06.762090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761116+0000","last_change":"2023-03-03T06:18:05.766192+0000","last_active":"2023-03-03T06:18:06.761116+0000","last_peered":"2023-03-03T06:18:06.761116+0000","last_clean":"2023-03-03T06:18:06.761116+0000","last_became_active":"2023-03-03T06:18:05.764744+0000","last_became_peered":"2023-03-03T06:18:05.764744+0000","last_unstale":"2023-03-03T06:18:06.761116+0000","last_undegraded":"2023-03-03T06:18:06.761116+0000","last_fullsized":"2023-03-03T06:18:06.761116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.776193+0000","last_change":"2023-03-03T06:18:05.378275+0000","last_active":"2023-03-03T06:18:05.776193+0000","last_peered":"2023-03-03T06:18:05.776193+0000","last_clean":"2023-03-03T06:18:05.776193+0000","last_became_active":"2023-03-03T06:18:05.377803+0000","last_became_peered":"2023-03-03T06:18:05.377803+0000","last_unstale":"2023-03-03T06:18:05.776193+0000","last_undegraded":"2023-03-03T06:18:05.776193+0000","last_fullsized":"2023-03-03T06:18:05.776193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780746+0000","last_change":"2023-03-03T06:18:05.770801+0000","last_active":"2023-03-03T06:18:05.780746+0000","last_peered":"2023-03-03T06:18:05.780746+0000","last_clean":"2023-03-03T06:18:05.780746+0000","last_became_active":"2023-03-03T06:18:05.770500+0000","last_became_peered":"2023-03-03T06:18:05.770500+0000","last_unstale":"2023-03-03T06:18:05.780746+0000","last_undegraded":"2023-03-03T06:18:05.780746+0000","last_fullsized":"2023-03-03T06:18:05.780746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.783309+0000","last_change":"2023-03-03T06:18:05.776954+0000","last_active":"2023-03-03T06:18:05.783309+0000","last_peered":"2023-03-03T06:18:05.783309+0000","last_clean":"2023-03-03T06:18:05.783309+0000","last_became_active":"2023-03-03T06:18:05.776665+0000","last_became_peered":"2023-03-03T06:18:05.776665+0000","last_unstale":"2023-03-03T06:18:05.783309+0000","last_undegraded":"2023-03-03T06:18:05.783309+0000","last_fullsized":"2023-03-03T06:18:05.783309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:46.917574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761055+0000","last_change":"2023-03-03T06:18:05.780029+0000","last_active":"2023-03-03T06:18:06.761055+0000","last_peered":"2023-03-03T06:18:06.761055+0000","last_clean":"2023-03-03T06:18:06.761055+0000","last_became_active":"2023-03-03T06:18:05.779805+0000","last_became_peered":"2023-03-03T06:18:05.779805+0000","last_unstale":"2023-03-03T06:18:06.761055+0000","last_undegraded":"2023-03-03T06:18:06.761055+0000","last_fullsized":"2023-03-03T06:18:06.761055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.784005+0000","last_change":"2023-03-03T06:18:05.781886+0000","last_active":"2023-03-03T06:18:05.784005+0000","last_peered":"2023-03-03T06:18:05.784005+0000","last_clean":"2023-03-03T06:18:05.784005+0000","last_became_active":"2023-03-03T06:18:05.780587+0000","last_became_peered":"2023-03-03T06:18:05.780587+0000","last_unstale":"2023-03-03T06:18:05.784005+0000","last_undegraded":"2023-03-03T06:18:05.784005+0000","last_fullsized":"2023-03-03T06:18:05.784005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:02.129741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761713+0000","last_change":"2023-03-03T06:18:05.760339+0000","last_active":"2023-03-03T06:18:06.761713+0000","last_peered":"2023-03-03T06:18:06.761713+0000","last_clean":"2023-03-03T06:18:06.761713+0000","last_became_active":"2023-03-03T06:18:05.759652+0000","last_became_peered":"2023-03-03T06:18:05.759652+0000","last_unstale":"2023-03-03T06:18:06.761713+0000","last_undegraded":"2023-03-03T06:18:06.761713+0000","last_fullsized":"2023-03-03T06:18:06.761713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.780979+0000","last_change":"2023-03-03T06:18:05.777608+0000","last_active":"2023-03-03T06:18:05.780979+0000","last_peered":"2023-03-03T06:18:05.780979+0000","last_clean":"2023-03-03T06:18:05.780979+0000","last_became_active":"2023-03-03T06:18:05.777310+0000","last_became_peered":"2023-03-03T06:18:05.777310+0000","last_unstale":"2023-03-03T06:18:05.780979+0000","last_undegraded":"2023-03-03T06:18:05.780979+0000","last_fullsized":"2023-03-03T06:18:05.780979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9096,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362744,"statfs":{"total":96636764160,"available":96627449856,"internally_reserved":0,"allocated":851968,"data_stored":675078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7976,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363864,"statfs":{"total":96636764160,"available":96628596736,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7976,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363864,"statfs":{"total":96636764160,"available":96628596736,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9072,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362768,"statfs":{"total":96636764160,"available":96627474432,"internally_reserved":0,"allocated":827392,"data_stored":673880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7912,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363928,"statfs":{"total":96636764160,"available":96628662272,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8044,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363796,"statfs":{"total":96636764160,"available":96628527104,"internally_reserved":0,"allocated":233472,"data_stored":83036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8060,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363780,"statfs":{"total":96636764160,"available":96628510720,"internally_reserved":0,"allocated":249856,"data_stored":84550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6040000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7050000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":83582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T06:18:10.196 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T06:18:10.197 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-03-03T06:18:10.197 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-03-03T06:18:10.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-03-03T06:18:10.198 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-03-03T06:18:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.349+0000 7f81f97ec700 1 Processor -- start 2023-03-03T06:18:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.349+0000 7f81f97ec700 1 -- start start 2023-03-03T06:18:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.349+0000 7f81f97ec700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f81f40c39b0 0x7f81f415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.349+0000 7f81f97ec700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f414fb40 con 0x7f81f406a330 2023-03-03T06:18:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.349+0000 7f81f97ec700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f406c7d0 con 0x7f81f406bcf0 2023-03-03T06:18:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.349+0000 7f81f97ec700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f40c0060 con 0x7f81f40c39b0 2023-03-03T06:18:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f37fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f81f40c39b0 0x7f81f415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f37fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f81f40c39b0 0x7f81f415aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34042/0 (socket says 172.21.15.90:34042) 2023-03-03T06:18:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f37fe700 1 -- 172.21.15.90:0/1929642625 learned_addr learned my addr 172.21.15.90:0/1929642625 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f37fe700 1 -- 172.21.15.90:0/1929642625 >> v1:172.21.15.121:6789/0 conn(0x7f81f406bcf0 legacy=0x7f81f406c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f37fe700 1 -- 172.21.15.90:0/1929642625 >> v1:172.21.15.90:6789/0 conn(0x7f81f406a330 legacy=0x7f81f40c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f37fe700 1 -- 172.21.15.90:0/1929642625 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f40b8d00 con 0x7f81f40c39b0 2023-03-03T06:18:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f37fe700 1 --2- 172.21.15.90:0/1929642625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f40c39b0 0x7f81f415aa50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f81ec009e50 tx=0x7f81ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54eb2164de0fb3e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f1ffb700 1 -- 172.21.15.90:0/1929642625 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec00e070 con 0x7f81f40c39b0 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f1ffb700 1 -- 172.21.15.90:0/1929642625 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ec003a20 con 0x7f81f40c39b0 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.350+0000 7f81f1ffb700 1 -- 172.21.15.90:0/1929642625 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec003dd0 con 0x7f81f40c39b0 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.351+0000 7f81f97ec700 1 -- 172.21.15.90:0/1929642625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f40c39b0 msgr2=0x7f81f415aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.351+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1929642625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f40c39b0 0x7f81f415aa50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f81ec009e50 tx=0x7f81ec00b330 comp rx=0 tx=0).stop 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.351+0000 7f81f97ec700 1 -- 172.21.15.90:0/1929642625 shutdown_connections 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.351+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1929642625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f40c39b0 0x7f81f415aa50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.355 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.351+0000 7f81f97ec700 1 -- 172.21.15.90:0/1929642625 >> 172.21.15.90:0/1929642625 conn(0x7f81f40a6f40 msgr2=0x7f81f40a7340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:10.356 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.351+0000 7f81f97ec700 1 -- 172.21.15.90:0/1929642625 shutdown_connections 2023-03-03T06:18:10.356 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.351+0000 7f81f97ec700 1 -- 172.21.15.90:0/1929642625 wait complete. 2023-03-03T06:18:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.352+0000 7f81f97ec700 1 Processor -- start 2023-03-03T06:18:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.352+0000 7f81f97ec700 1 -- start start 2023-03-03T06:18:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f406a330 0x7f81f40bd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f406bcf0 0x7f81f40a9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81f40c39b0 0x7f81f414bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40a6890 con 0x7f81f406bcf0 2023-03-03T06:18:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40a9dc0 con 0x7f81f406a330 2023-03-03T06:18:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40aaa30 con 0x7f81f40c39b0 2023-03-03T06:18:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f406bcf0 0x7f81f40a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f406bcf0 0x7f81f40a9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34046/0 (socket says 172.21.15.90:34046) 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 -- 172.21.15.90:0/1875148437 learned_addr learned my addr 172.21.15.90:0/1875148437 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f2ffd700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f406a330 0x7f81f40bd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f37fe700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81f40c39b0 0x7f81f414bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 -- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81f40c39b0 msgr2=0x7f81f414bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81f40c39b0 0x7f81f414bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 -- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f406a330 msgr2=0x7f81f40bd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f406a330 0x7f81f40bd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.360 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 -- 172.21.15.90:0/1875148437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec009b30 con 0x7f81f406bcf0 2023-03-03T06:18:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f27fc700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f406bcf0 0x7f81f40a9880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81e800c430 tx=0x7f81e800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81db7fe700 1 -- 172.21.15.90:0/1875148437 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e80050d0 con 0x7f81f406bcf0 2023-03-03T06:18:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81db7fe700 1 -- 172.21.15.90:0/1875148437 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e8005250 con 0x7f81f406bcf0 2023-03-03T06:18:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f414fd90 con 0x7f81f406bcf0 2023-03-03T06:18:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81db7fe700 1 -- 172.21.15.90:0/1875148437 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8006860 con 0x7f81f406bcf0 2023-03-03T06:18:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.353+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f40c0300 con 0x7f81f406bcf0 2023-03-03T06:18:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.354+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f40c51b0 con 0x7f81f406bcf0 2023-03-03T06:18:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.355+0000 7f81db7fe700 1 -- 172.21.15.90:0/1875148437 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81e8010050 con 0x7f81f406bcf0 2023-03-03T06:18:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.355+0000 7f81db7fe700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81e00682f0 0x7f81e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.355+0000 7f81db7fe700 1 -- 172.21.15.90:0/1875148437 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f81e8084bf0 con 0x7f81f406bcf0 2023-03-03T06:18:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.355+0000 7f81f2ffd700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81e00682f0 0x7f81e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.355+0000 7f81f2ffd700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81e00682f0 0x7f81e006a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009960 tx=0x7f81e40090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:10.363 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.359+0000 7f81db7fe700 1 -- 172.21.15.90:0/1875148437 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81e804a5e0 con 0x7f81f406bcf0 2023-03-03T06:18:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.593+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f81f40aac00 con 0x7f81f406bcf0 2023-03-03T06:18:10.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.596+0000 7f81db7fe700 1 -- 172.21.15.90:0/1875148437 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+166374 (secure 0 0 0) 0x7f81e8048a00 con 0x7f81f406bcf0 2023-03-03T06:18:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:10.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81e00682f0 msgr2=0x7f81e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81e00682f0 0x7f81e006a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009960 tx=0x7f81e40090e0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f406bcf0 msgr2=0x7f81f40a9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f406bcf0 0x7f81f40a9880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81e800c430 tx=0x7f81e800c740 comp rx=0 tx=0).stop 2023-03-03T06:18:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 shutdown_connections 2023-03-03T06:18:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81f40c39b0 0x7f81f414bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f406a330 0x7f81f40bd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81e00682f0 0x7f81e006a7a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 --2- 172.21.15.90:0/1875148437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81f406bcf0 0x7f81f40a9880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 >> 172.21.15.90:0/1875148437 conn(0x7f81f40a6f40 msgr2=0x7f81f40b6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 shutdown_connections 2023-03-03T06:18:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.598+0000 7f81f97ec700 1 -- 172.21.15.90:0/1875148437 wait complete. 2023-03-03T06:18:10.612 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"active_gid":4108,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":96475},{"type":"v1","addr":"172.21.15.90:6833","nonce":96475}]},"active_addr":"172.21.15.90:6833/96475","active_change":"2023-03-03T06:17:46.066161+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1336332253}]},{"addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":29046459}]},{"addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":3415430726}]},{"addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2826878421}]}]} 2023-03-03T06:18:10.617 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-03-03T06:18:10.618 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-03-03T06:18:10.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.735+0000 7fb358971700 1 Processor -- start 2023-03-03T06:18:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.735+0000 7fb358971700 1 -- start start 2023-03-03T06:18:10.739 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb358971700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb354145970 0x7fb3540bd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.739 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb358971700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb3540bdd00 0x7fb354155e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.739 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb358971700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35406dd10 con 0x7fb3540bcf00 2023-03-03T06:18:10.739 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb358971700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3540c15a0 con 0x7fb3540bdd00 2023-03-03T06:18:10.739 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb358971700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3540bc380 con 0x7fb354145970 2023-03-03T06:18:10.739 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb3540bdd00 0x7fb354155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb3540bdd00 0x7fb354155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34056/0 (socket says 172.21.15.90:34056) 2023-03-03T06:18:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 -- 172.21.15.90:0/4101084191 learned_addr learned my addr 172.21.15.90:0/4101084191 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb35259c700 1 --2- 172.21.15.90:0/4101084191 >> v2:172.21.15.121:3300/0 conn(0x7fb354145970 0x7fb3540bd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 -- 172.21.15.90:0/4101084191 >> v2:172.21.15.121:3300/0 conn(0x7fb354145970 msgr2=0x7fb3540bd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 --2- 172.21.15.90:0/4101084191 >> v2:172.21.15.121:3300/0 conn(0x7fb354145970 0x7fb3540bd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 -- 172.21.15.90:0/4101084191 >> v1:172.21.15.121:6789/0 conn(0x7fb3540bcf00 legacy=0x7fb35414ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 -- 172.21.15.90:0/4101084191 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3540c1300 con 0x7fb3540bdd00 2023-03-03T06:18:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.736+0000 7fb352d9d700 1 --2- 172.21.15.90:0/4101084191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb354155e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb35406e350 tx=0x7fb34c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab21b061b52f013 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb35159a700 1 -- 172.21.15.90:0/4101084191 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00e070 con 0x7fb3540bdd00 2023-03-03T06:18:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb35159a700 1 -- 172.21.15.90:0/4101084191 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c00b910 con 0x7fb3540bdd00 2023-03-03T06:18:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb35159a700 1 -- 172.21.15.90:0/4101084191 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00bcc0 con 0x7fb3540bdd00 2023-03-03T06:18:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 -- 172.21.15.90:0/4101084191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 msgr2=0x7fb354155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 --2- 172.21.15.90:0/4101084191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb354155e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb35406e350 tx=0x7fb34c00b330 comp rx=0 tx=0).stop 2023-03-03T06:18:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 -- 172.21.15.90:0/4101084191 shutdown_connections 2023-03-03T06:18:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 --2- 172.21.15.90:0/4101084191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb354155e60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 --2- 172.21.15.90:0/4101084191 >> v2:172.21.15.121:3300/0 conn(0x7fb354145970 0x7fb3540bd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 -- 172.21.15.90:0/4101084191 >> 172.21.15.90:0/4101084191 conn(0x7fb3540c1900 msgr2=0x7fb3540c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 -- 172.21.15.90:0/4101084191 shutdown_connections 2023-03-03T06:18:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.737+0000 7fb358971700 1 -- 172.21.15.90:0/4101084191 wait complete. 2023-03-03T06:18:10.744 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.738+0000 7fb358971700 1 Processor -- start 2023-03-03T06:18:10.744 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.738+0000 7fb358971700 1 -- start start 2023-03-03T06:18:10.744 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb358971700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3540bcf00 0x7fb354147030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb358971700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb3541e3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb358971700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb354145970 0x7fb3540be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb358971700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3540be910 con 0x7fb3540bdd00 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb358971700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3541e27c0 con 0x7fb354145970 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb358971700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35414bb20 con 0x7fb3540bcf00 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb3541e3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb3541e3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34060/0 (socket says 172.21.15.90:34060) 2023-03-03T06:18:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 -- 172.21.15.90:0/2529421451 learned_addr learned my addr 172.21.15.90:0/2529421451 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb35259c700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3540bcf00 0x7fb354147030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb352d9d700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb354145970 0x7fb3540be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 -- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3540bcf00 msgr2=0x7fb354147030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3540bcf00 0x7fb354147030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 -- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb354145970 msgr2=0x7fb3540be3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb354145970 0x7fb3540be3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.746 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.739+0000 7fb351d9b700 1 -- 172.21.15.90:0/2529421451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c009b30 con 0x7fb3540bdd00 2023-03-03T06:18:10.747 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.740+0000 7fb351d9b700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb3541e3280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb34400c4e0 tx=0x7fb34400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:10.747 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.740+0000 7fb342ffd700 1 -- 172.21.15.90:0/2529421451 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3440052d0 con 0x7fb3540bdd00 2023-03-03T06:18:10.747 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.740+0000 7fb342ffd700 1 -- 172.21.15.90:0/2529421451 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb344005430 con 0x7fb3540bdd00 2023-03-03T06:18:10.747 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.740+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3540a0e60 con 0x7fb3540bdd00 2023-03-03T06:18:10.747 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.740+0000 7fb342ffd700 1 -- 172.21.15.90:0/2529421451 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3440038c0 con 0x7fb3540bdd00 2023-03-03T06:18:10.747 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.740+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35406da30 con 0x7fb3540bdd00 2023-03-03T06:18:10.747 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.741+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3540a1140 con 0x7fb3540bdd00 2023-03-03T06:18:10.748 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.741+0000 7fb342ffd700 1 -- 172.21.15.90:0/2529421451 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb344019030 con 0x7fb3540bdd00 2023-03-03T06:18:10.748 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.742+0000 7fb342ffd700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb3340682f0 0x7fb33406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:10.748 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.742+0000 7fb342ffd700 1 -- 172.21.15.90:0/2529421451 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb344084ef0 con 0x7fb3540bdd00 2023-03-03T06:18:10.748 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.742+0000 7fb35259c700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb3340682f0 0x7fb33406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:10.748 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.742+0000 7fb35259c700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb3340682f0 0x7fb33406a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb35414c880 tx=0x7fb34800a320 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:10.749 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.746+0000 7fb342ffd700 1 -- 172.21.15.90:0/2529421451 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb344051750 con 0x7fb3540bdd00 2023-03-03T06:18:10.930 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.928+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb35414c040 con 0x7fb3540bdd00 2023-03-03T06:18:10.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.930+0000 7fb342ffd700 1 -- 172.21.15.90:0/2529421451 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17811 (secure 0 0 0) 0x7fb344017090 con 0x7fb3540bdd00 2023-03-03T06:18:10.932 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:10.932 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":23,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:06.754461+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T06:18:03.400568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T06:18:04.718088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:10.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb3340682f0 msgr2=0x7fb33406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb3340682f0 0x7fb33406a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb35414c880 tx=0x7fb34800a320 comp rx=0 tx=0).stop 2023-03-03T06:18:10.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 msgr2=0x7fb3541e3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:10.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb3541e3280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb34400c4e0 tx=0x7fb34400c7f0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 shutdown_connections 2023-03-03T06:18:10.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb354145970 0x7fb3540be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3540bcf00 0x7fb354147030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb3340682f0 0x7fb33406a7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 --2- 172.21.15.90:0/2529421451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3540bdd00 0x7fb3541e3280 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:10.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 >> 172.21.15.90:0/2529421451 conn(0x7fb3540c1900 msgr2=0x7fb354144f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:10.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 shutdown_connections 2023-03-03T06:18:10.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:10.932+0000 7fb358971700 1 -- 172.21.15.90:0/2529421451 wait complete. 2023-03-03T06:18:10.946 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-03-03T06:18:10.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:11.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.063+0000 7f50746ba700 1 Processor -- start 2023-03-03T06:18:11.068 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.063+0000 7f50746ba700 1 -- start start 2023-03-03T06:18:11.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f50746ba700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f506c0be4e0 0x7f506c151040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f50746ba700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f50746ba700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c06ed70 con 0x7f506c072420 2023-03-03T06:18:11.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f50746ba700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506c0713a0 con 0x7f506c0be4e0 2023-03-03T06:18:11.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f50746ba700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c06d420 con 0x7f506c0a4d00 2023-03-03T06:18:11.069 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5071c55700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5072456700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f506c0be4e0 0x7f506c151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5072c57700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f506c072420 0x7f506c0c2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:45918/0 (socket says 172.21.15.90:45918) 2023-03-03T06:18:11.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5072c57700 1 -- 172.21.15.90:0/3305913601 learned_addr learned my addr 172.21.15.90:0/3305913601 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:11.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5071454700 1 -- 172.21.15.90:0/3305913601 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993885902 0 0) 0x7f506c06ed70 con 0x7f506c072420 2023-03-03T06:18:11.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5071454700 1 -- 172.21.15.90:0/3305913601 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50580035b0 con 0x7f506c072420 2023-03-03T06:18:11.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5071c55700 1 -- 172.21.15.90:0/3305913601 >> v2:172.21.15.121:3300/0 conn(0x7f506c0be4e0 msgr2=0x7f506c151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.070 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5071c55700 1 --2- 172.21.15.90:0/3305913601 >> v2:172.21.15.121:3300/0 conn(0x7f506c0be4e0 0x7f506c151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.064+0000 7f5071c55700 1 -- 172.21.15.90:0/3305913601 >> v1:172.21.15.121:6789/0 conn(0x7f506c072420 legacy=0x7f506c0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f5071c55700 1 -- 172.21.15.90:0/3305913601 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c06ea90 con 0x7f506c0a4d00 2023-03-03T06:18:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f5072456700 1 --2- 172.21.15.90:0/3305913601 >> v2:172.21.15.121:3300/0 conn(0x7f506c0be4e0 0x7f506c151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f5071c55700 1 --2- 172.21.15.90:0/3305913601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c14ced0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5068009840 tx=0x7f5068009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d40558bf05f8ce2e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f5071454700 1 -- 172.21.15.90:0/3305913601 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068011830 con 0x7f506c0a4d00 2023-03-03T06:18:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f5071454700 1 -- 172.21.15.90:0/3305913601 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5068011990 con 0x7f506c0a4d00 2023-03-03T06:18:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f5071454700 1 -- 172.21.15.90:0/3305913601 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068011d20 con 0x7f506c0a4d00 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f50746ba700 1 -- 172.21.15.90:0/3305913601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 msgr2=0x7f506c14ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f50746ba700 1 --2- 172.21.15.90:0/3305913601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c14ced0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5068009840 tx=0x7f5068009b50 comp rx=0 tx=0).stop 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.065+0000 7f50746ba700 1 -- 172.21.15.90:0/3305913601 shutdown_connections 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.066+0000 7f50746ba700 1 --2- 172.21.15.90:0/3305913601 >> v2:172.21.15.121:3300/0 conn(0x7f506c0be4e0 0x7f506c151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.066+0000 7f50746ba700 1 --2- 172.21.15.90:0/3305913601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c14ced0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.066+0000 7f50746ba700 1 -- 172.21.15.90:0/3305913601 >> 172.21.15.90:0/3305913601 conn(0x7f506c06c0d0 msgr2=0x7f506c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.066+0000 7f50746ba700 1 -- 172.21.15.90:0/3305913601 shutdown_connections 2023-03-03T06:18:11.072 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.066+0000 7f50746ba700 1 -- 172.21.15.90:0/3305913601 wait complete. 2023-03-03T06:18:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 Processor -- start 2023-03-03T06:18:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 -- start start 2023-03-03T06:18:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c0a7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f506c0be4e0 0x7f506c0bcf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f506c141ec0 0x7f506c1408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c147210 con 0x7f506c141ec0 2023-03-03T06:18:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0bd460 con 0x7f506c0be4e0 2023-03-03T06:18:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f50746ba700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c1e3260 con 0x7f506c0a4d00 2023-03-03T06:18:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f5072456700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c0a7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f5072456700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c0a7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39006/0 (socket says 172.21.15.90:39006) 2023-03-03T06:18:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f5072456700 1 -- 172.21.15.90:0/2408052610 learned_addr learned my addr 172.21.15.90:0/2408052610 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f5072c57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f506c141ec0 0x7f506c1408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.067+0000 7f5071c55700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f506c0be4e0 0x7f506c0bcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5072456700 1 -- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f506c0be4e0 msgr2=0x7f506c0bcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5072456700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f506c0be4e0 0x7f506c0bcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5072456700 1 -- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f506c141ec0 msgr2=0x7f506c1408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5072456700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f506c141ec0 0x7f506c1408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5072456700 1 -- 172.21.15.90:0/2408052610 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c0094a0 con 0x7f506c0a4d00 2023-03-03T06:18:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5072456700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c0a7df0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f505c00ec10 tx=0x7f505c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5062ffd700 1 -- 172.21.15.90:0/2408052610 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c009760 con 0x7f506c0a4d00 2023-03-03T06:18:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5062ffd700 1 -- 172.21.15.90:0/2408052610 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c00c850 con 0x7f506c0a4d00 2023-03-03T06:18:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50680094a0 con 0x7f506c0a4d00 2023-03-03T06:18:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f5062ffd700 1 -- 172.21.15.90:0/2408052610 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c00cbe0 con 0x7f506c0a4d00 2023-03-03T06:18:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.068+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c150840 con 0x7f506c0a4d00 2023-03-03T06:18:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.069+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c150560 con 0x7f506c0a4d00 2023-03-03T06:18:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.070+0000 7f5062ffd700 1 -- 172.21.15.90:0/2408052610 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f505c019030 con 0x7f506c0a4d00 2023-03-03T06:18:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.070+0000 7f5062ffd700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f5050068320 0x7f505006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.070+0000 7f5062ffd700 1 -- 172.21.15.90:0/2408052610 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f505c085bb0 con 0x7f506c0a4d00 2023-03-03T06:18:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.070+0000 7f5071c55700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f5050068320 0x7f505006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.070+0000 7f5071c55700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f5050068320 0x7f505006a7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5068009840 tx=0x7f5068008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.075+0000 7f5062ffd700 1 -- 172.21.15.90:0/2408052610 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f505c052380 con 0x7f506c0a4d00 2023-03-03T06:18:11.262 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.261+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f506c150b20 con 0x7f506c0a4d00 2023-03-03T06:18:11.264 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.262+0000 7f5062ffd700 1 -- 172.21.15.90:0/2408052610 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17811 (secure 0 0 0) 0x7f505c021090 con 0x7f506c0a4d00 2023-03-03T06:18:11.264 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:11.265 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":23,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:06.754461+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T06:18:03.400568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T06:18:04.718088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f5050068320 msgr2=0x7f505006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f5050068320 0x7f505006a7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5068009840 tx=0x7f5068008030 comp rx=0 tx=0).stop 2023-03-03T06:18:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 msgr2=0x7f506c0a7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c0a7df0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f505c00ec10 tx=0x7f505c00ef20 comp rx=0 tx=0).stop 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 shutdown_connections 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f5050068320 0x7f505006a7d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f506c141ec0 0x7f506c1408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f506c0a4d00 0x7f506c0a7df0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 --2- 172.21.15.90:0/2408052610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f506c0be4e0 0x7f506c0bcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.264+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 >> 172.21.15.90:0/2408052610 conn(0x7f506c06c0d0 msgr2=0x7f506c13fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.265+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 shutdown_connections 2023-03-03T06:18:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.265+0000 7f50746ba700 1 -- 172.21.15.90:0/2408052610 wait complete. 2023-03-03T06:18:11.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-03-03T06:18:11.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-03-03T06:18:11.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-03-03T06:18:11.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-03-03T06:18:11.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-03-03T06:18:11.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-03-03T06:18:11.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-03-03T06:18:11.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-03-03T06:18:11.476 INFO:tasks.ceph.mds.c.smithi090.stderr:2023-03-03T06:18:11.473+0000 7f3349450700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T06:18:11.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.487+0000 7f05deee1700 1 Processor -- start 2023-03-03T06:18:11.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.487+0000 7f05deee1700 1 -- start start 2023-03-03T06:18:11.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.488+0000 7f05deee1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05d8145510 0x7f05d814f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.488+0000 7f05deee1700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d80bc4a0 con 0x7f05d80b5fd0 2023-03-03T06:18:11.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.488+0000 7f05deee1700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d81458e0 con 0x7f05d80bd3f0 2023-03-03T06:18:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.488+0000 7f05deee1700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d80b7e40 con 0x7f05d8145510 2023-03-03T06:18:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.489+0000 7f05dcc7d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05d8145510 0x7f05d814f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.489+0000 7f05dcc7d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05d8145510 0x7f05d814f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48220/0 (socket says 172.21.15.90:48220) 2023-03-03T06:18:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.489+0000 7f05dcc7d700 1 -- 172.21.15.90:0/4081023684 learned_addr learned my addr 172.21.15.90:0/4081023684 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.489+0000 7f05b9a1d700 1 Processor -- start 2023-03-03T06:18:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.489+0000 7f05b9a1d700 1 -- start start 2023-03-03T06:18:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05dcc7d700 1 -- 172.21.15.90:0/4081023684 >> v1:172.21.15.121:6789/0 conn(0x7f05d80bd3f0 legacy=0x7f05d8155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05dcc7d700 1 -- 172.21.15.90:0/4081023684 >> v1:172.21.15.90:6789/0 conn(0x7f05d80b5fd0 legacy=0x7f05d806c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b9a1d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05b41452f0 0x7f05b414f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b9a1d700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f05b4153b00 0x7f05b4155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b9a1d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b41456c0 con 0x7f05b40bcba0 2023-03-03T06:18:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b9a1d700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b414eb90 con 0x7f05b4153b00 2023-03-03T06:18:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b9a1d700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b414bdb0 con 0x7f05b41452f0 2023-03-03T06:18:11.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05dcc7d700 1 -- 172.21.15.90:0/4081023684 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d80bbb70 con 0x7f05d8145510 2023-03-03T06:18:11.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b37fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f05b4153b00 0x7f05b4155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b37fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f05b4153b00 0x7f05b4155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34084/0 (socket says 172.21.15.90:34084) 2023-03-03T06:18:11.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b37fe700 1 -- 172.21.15.90:0/3382869438 learned_addr learned my addr 172.21.15.90:0/3382869438 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b2ffd700 1 --2- 172.21.15.90:0/3382869438 >> v2:172.21.15.121:3300/0 conn(0x7f05b41452f0 0x7f05b414f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b37fe700 1 -- 172.21.15.90:0/3382869438 >> v2:172.21.15.121:3300/0 conn(0x7f05b41452f0 msgr2=0x7f05b414f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.490+0000 7f05b37fe700 1 --2- 172.21.15.90:0/3382869438 >> v2:172.21.15.121:3300/0 conn(0x7f05b41452f0 0x7f05b414f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7f05b37fe700 1 -- 172.21.15.90:0/3382869438 >> v1:172.21.15.121:6789/0 conn(0x7f05b40bcba0 legacy=0x7f05b40c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7f05b37fe700 1 -- 172.21.15.90:0/3382869438 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b414e820 con 0x7f05b4153b00 2023-03-03T06:18:11.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7fa6a778f700 1 Processor -- start 2023-03-03T06:18:11.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7f05b37fe700 1 --2- 172.21.15.90:0/3382869438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b4153b00 0x7f05b4155ee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05b40bc250 tx=0x7f05ac007890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6edc6100d19400c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7fa6a778f700 1 -- start start 2023-03-03T06:18:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7f05b1ffb700 1 -- 172.21.15.90:0/3382869438 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac00ce60 con 0x7f05b4153b00 2023-03-03T06:18:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7f05b1ffb700 1 -- 172.21.15.90:0/3382869438 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ac003a20 con 0x7f05b4153b00 2023-03-03T06:18:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.491+0000 7f05b1ffb700 1 -- 172.21.15.90:0/3382869438 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac003dd0 con 0x7f05b4153b00 2023-03-03T06:18:11.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05dcc7d700 1 --2- 172.21.15.90:0/4081023684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d8145510 0x7f05d814f0f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009730 tx=0x7f05d0009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a42fd7c5f145814 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a778f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa6a00c19d0 0x7fa6a00bc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a778f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd630 0x7fa6a00b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a778f700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa6a00bd1e0 0x7fa6a014c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a778f700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a00b6ef0 con 0x7fa6a00bd1e0 2023-03-03T06:18:11.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a778f700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a014fe00 con 0x7fa6a00c19d0 2023-03-03T06:18:11.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a778f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00b7190 con 0x7fa6a00bd630 2023-03-03T06:18:11.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05b9a1d700 1 -- 172.21.15.90:0/3382869438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b4153b00 msgr2=0x7f05b4155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05b9a1d700 1 --2- 172.21.15.90:0/3382869438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b4153b00 0x7f05b4155ee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05b40bc250 tx=0x7f05ac007890 comp rx=0 tx=0).stop 2023-03-03T06:18:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a5d2c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa6a00bd1e0 0x7fa6a014c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a5d2c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa6a00bd1e0 0x7fa6a014c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34094/0 (socket says 172.21.15.90:34094) 2023-03-03T06:18:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a5d2c700 1 -- 172.21.15.90:0/2868562555 learned_addr learned my addr 172.21.15.90:0/2868562555 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a552b700 1 --2- 172.21.15.90:0/2868562555 >> v2:172.21.15.121:3300/0 conn(0x7fa6a00c19d0 0x7fa6a00bc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7fa6a4d2a700 1 --2- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd630 0x7fa6a00b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05b9a1d700 1 -- 172.21.15.90:0/3382869438 shutdown_connections 2023-03-03T06:18:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05b9a1d700 1 --2- 172.21.15.90:0/3382869438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b4153b00 0x7f05b4155ee0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05b9a1d700 1 --2- 172.21.15.90:0/3382869438 >> v2:172.21.15.121:3300/0 conn(0x7f05b41452f0 0x7f05b414f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05b9a1d700 1 -- 172.21.15.90:0/3382869438 >> 172.21.15.90:0/3382869438 conn(0x7f05b406afc0 msgr2=0x7f05b40c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05d77fe700 1 -- 172.21.15.90:0/4081023684 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d000c8f0 con 0x7f05d8145510 2023-03-03T06:18:11.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05d77fe700 1 -- 172.21.15.90:0/4081023684 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d000ca50 con 0x7f05d8145510 2023-03-03T06:18:11.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7f05d77fe700 1 -- 172.21.15.90:0/4081023684 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d000ce00 con 0x7f05d8145510 2023-03-03T06:18:11.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7f05b9a1d700 1 -- 172.21.15.90:0/3382869438 shutdown_connections 2023-03-03T06:18:11.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7f05b9a1d700 1 -- 172.21.15.90:0/3382869438 wait complete. 2023-03-03T06:18:11.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fa6a552b700 1 -- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd630 msgr2=0x7fa6a00b62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fa6a552b700 1 --2- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd630 0x7fa6a00b62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fa6a552b700 1 -- 172.21.15.90:0/2868562555 >> v2:172.21.15.90:3300/0 conn(0x7fa6a00bd1e0 msgr2=0x7fa6a014c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fa6a552b700 1 --2- 172.21.15.90:0/2868562555 >> v2:172.21.15.90:3300/0 conn(0x7fa6a00bd1e0 0x7fa6a014c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fa6a552b700 1 -- 172.21.15.90:0/2868562555 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00b8660 con 0x7fa6a00c19d0 2023-03-03T06:18:11.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7feddd3d2700 1 Processor -- start 2023-03-03T06:18:11.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.492+0000 7feddd3d2700 1 -- start start 2023-03-03T06:18:11.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7feddd3d2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80c19d0 0x7fedd806c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7feddd3d2700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fedd80a6e80 0x7fedd815de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7feddd3d2700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd80bd250 con 0x7fedd80bd6e0 2023-03-03T06:18:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7feddd3d2700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd81542d0 con 0x7fedd80a6e80 2023-03-03T06:18:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7feddd3d2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80b6360 con 0x7fedd80c19d0 2023-03-03T06:18:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fedd77fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fedd80a6e80 0x7fedd815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fedd77fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fedd80a6e80 0x7fedd815de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34098/0 (socket says 172.21.15.90:34098) 2023-03-03T06:18:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fedd77fe700 1 -- 172.21.15.90:0/3455950844 learned_addr learned my addr 172.21.15.90:0/3455950844 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fa6a552b700 1 --2- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 0x7fa6a00bc1f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00b78e0 tx=0x7fa690009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=959935d424d77046 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7f05deee1700 1 -- 172.21.15.90:0/4081023684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d8145510 msgr2=0x7f05d814f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7f05deee1700 1 --2- 172.21.15.90:0/4081023684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d8145510 0x7f05d814f0f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009730 tx=0x7f05d0009760 comp rx=0 tx=0).stop 2023-03-03T06:18:11.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.493+0000 7fa697fff700 1 -- 172.21.15.90:0/2868562555 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690011830 con 0x7fa6a00c19d0 2023-03-03T06:18:11.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fa697fff700 1 -- 172.21.15.90:0/2868562555 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa690011990 con 0x7fa6a00c19d0 2023-03-03T06:18:11.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fa697fff700 1 -- 172.21.15.90:0/2868562555 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690011d00 con 0x7fa6a00c19d0 2023-03-03T06:18:11.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7f05deee1700 1 -- 172.21.15.90:0/4081023684 shutdown_connections 2023-03-03T06:18:11.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7f05deee1700 1 --2- 172.21.15.90:0/4081023684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d8145510 0x7f05d814f0f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7f05deee1700 1 -- 172.21.15.90:0/4081023684 >> 172.21.15.90:0/4081023684 conn(0x7f05d806afc0 msgr2=0x7f05d80c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd77fe700 1 -- 172.21.15.90:0/3455950844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80c19d0 msgr2=0x7fedd806c8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T06:18:11.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd77fe700 1 --2- 172.21.15.90:0/3455950844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80c19d0 0x7fedd806c8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd77fe700 1 -- 172.21.15.90:0/3455950844 >> v1:172.21.15.90:6789/0 conn(0x7fedd80bd6e0 legacy=0x7fedd80baa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd77fe700 1 -- 172.21.15.90:0/3455950844 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80b6640 con 0x7fedd80a6e80 2023-03-03T06:18:11.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd77fe700 1 --2- 172.21.15.90:0/3455950844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 0x7fedd815de10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fedc80098f0 tx=0x7fedc8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c300258972fb5a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd5ffb700 1 -- 172.21.15.90:0/3455950844 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800b8a0 con 0x7fedd80a6e80 2023-03-03T06:18:11.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7f05b9a1d700 1 Processor -- start 2023-03-03T06:18:11.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd5ffb700 1 -- 172.21.15.90:0/3455950844 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc800ba00 con 0x7fedd80a6e80 2023-03-03T06:18:11.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fedd5ffb700 1 -- 172.21.15.90:0/3455950844 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800bd90 con 0x7fedd80a6e80 2023-03-03T06:18:11.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7f05b9a1d700 1 -- start start 2023-03-03T06:18:11.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fa6a778f700 1 -- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 msgr2=0x7fa6a00bc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7fa6a778f700 1 --2- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 0x7fa6a00bc1f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00b78e0 tx=0x7fa690009b20 comp rx=0 tx=0).stop 2023-03-03T06:18:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.494+0000 7f05deee1700 1 -- 172.21.15.90:0/4081023684 shutdown_connections 2023-03-03T06:18:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.495+0000 7f05deee1700 1 -- 172.21.15.90:0/4081023684 wait complete. 2023-03-03T06:18:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.495+0000 7f05b9a1d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b40bcba0 0x7f05b4071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.495+0000 7f05b9a1d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b41452f0 0x7f05b41e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.495+0000 7f05b9a1d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05b4153b00 0x7f05b4147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.495+0000 7f05b9a1d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b41e9160 con 0x7f05b40bcba0 2023-03-03T06:18:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.495+0000 7f05b9a1d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b40bafe0 con 0x7f05b41452f0 2023-03-03T06:18:11.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.495+0000 7f05b9a1d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b40a6b10 con 0x7f05b4153b00 2023-03-03T06:18:11.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b40bcba0 0x7f05b4071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b40bcba0 0x7f05b4071d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34100/0 (socket says 172.21.15.90:34100) 2023-03-03T06:18:11.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 -- 172.21.15.90:0/283804299 learned_addr learned my addr 172.21.15.90:0/283804299 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05deee1700 1 Processor -- start 2023-03-03T06:18:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b27fc700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b41452f0 0x7f05b41e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b37fe700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05b4153b00 0x7f05b4147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 -- 172.21.15.90:0/283804299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05b4153b00 msgr2=0x7f05b4147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05b4153b00 0x7f05b4147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 -- 172.21.15.90:0/283804299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b41452f0 msgr2=0x7f05b41e95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b41452f0 0x7f05b41e95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 -- 172.21.15.90:0/283804299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac007210 con 0x7f05b40bcba0 2023-03-03T06:18:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05b2ffd700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b40bcba0 0x7f05b4071d30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f05a400dbd0 tx=0x7f05a400dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f059bfff700 1 -- 172.21.15.90:0/283804299 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40089f0 con 0x7f05b40bcba0 2023-03-03T06:18:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f059bfff700 1 -- 172.21.15.90:0/283804299 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a4008b50 con 0x7f05b40bcba0 2023-03-03T06:18:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f059bfff700 1 -- 172.21.15.90:0/283804299 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40153f0 con 0x7f05b40bcba0 2023-03-03T06:18:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b40c1e70 con 0x7f05b40bcba0 2023-03-03T06:18:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b40bc720 con 0x7f05b40bcba0 2023-03-03T06:18:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.496+0000 7f05deee1700 1 -- start start 2023-03-03T06:18:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05deee1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d80b5fd0 0x7f05d806f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05deee1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d80bd3f0 0x7f05d814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05deee1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05d8145510 0x7f05d814bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05deee1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d81485b0 con 0x7f05d8145510 2023-03-03T06:18:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05deee1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d806fc70 con 0x7f05d80bd3f0 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05deee1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d81482d0 con 0x7f05d80b5fd0 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d80b5fd0 0x7f05d806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d80b5fd0 0x7f05d806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39042/0 (socket says 172.21.15.90:39042) 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 -- 172.21.15.90:0/741191565 learned_addr learned my addr 172.21.15.90:0/741191565 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05d7fff700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d80bd3f0 0x7f05d814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dd47e700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05d8145510 0x7f05d814bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 -- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d80bd3f0 msgr2=0x7f05d814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05d80bd3f0 0x7f05d814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 -- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05d8145510 msgr2=0x7f05d814bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05d8145510 0x7f05d814bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dcc7d700 1 -- 172.21.15.90:0/741191565 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d000c3d0 con 0x7f05d80b5fd0 2023-03-03T06:18:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7f05dd47e700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05d8145510 0x7f05d814bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7feddd3d2700 1 -- 172.21.15.90:0/3455950844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 msgr2=0x7fedd815de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.497+0000 7feddd3d2700 1 --2- 172.21.15.90:0/3455950844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 0x7fedd815de10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fedc80098f0 tx=0x7fedc8009c00 comp rx=0 tx=0).stop 2023-03-03T06:18:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7feddd3d2700 1 -- 172.21.15.90:0/3455950844 shutdown_connections 2023-03-03T06:18:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7feddd3d2700 1 --2- 172.21.15.90:0/3455950844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 0x7fedd815de10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7feddd3d2700 1 --2- 172.21.15.90:0/3455950844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80c19d0 0x7fedd806c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7feddd3d2700 1 -- 172.21.15.90:0/3455950844 >> 172.21.15.90:0/3455950844 conn(0x7fedd806afc0 msgr2=0x7fedd80c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7f05dcc7d700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d80b5fd0 0x7f05d806f730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f05d00097a0 tx=0x7f05d0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7f05d57fa700 1 -- 172.21.15.90:0/741191565 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0003e30 con 0x7f05d80b5fd0 2023-03-03T06:18:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7f05d57fa700 1 -- 172.21.15.90:0/741191565 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d00092e0 con 0x7f05d80b5fd0 2023-03-03T06:18:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7f05d57fa700 1 -- 172.21.15.90:0/741191565 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0007b50 con 0x7f05d80b5fd0 2023-03-03T06:18:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7fa6a778f700 1 -- 172.21.15.90:0/2868562555 shutdown_connections 2023-03-03T06:18:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7fa6a778f700 1 --2- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 0x7fa6a00bc1f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7fa6a778f700 1 --2- 172.21.15.90:0/2868562555 >> v2:172.21.15.90:3300/0 conn(0x7fa6a00bd1e0 0x7fa6a014c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7fa6a778f700 1 --2- 172.21.15.90:0/2868562555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd630 0x7fa6a00b62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7fa6a778f700 1 -- 172.21.15.90:0/2868562555 >> 172.21.15.90:0/2868562555 conn(0x7fa6a006afc0 msgr2=0x7fa6a00c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.499+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d80bc130 con 0x7f05d80b5fd0 2023-03-03T06:18:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.499+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d80bbe50 con 0x7f05d80b5fd0 2023-03-03T06:18:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.499+0000 7f059bfff700 1 -- 172.21.15.90:0/283804299 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f05a401c030 con 0x7f05b40bcba0 2023-03-03T06:18:11.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7fa6a778f700 1 -- 172.21.15.90:0/2868562555 shutdown_connections 2023-03-03T06:18:11.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7fa6a778f700 1 -- 172.21.15.90:0/2868562555 wait complete. 2023-03-03T06:18:11.524 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7f059bfff700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05a00682f0 0x7f05a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7fea779d7700 1 Processor -- start 2023-03-03T06:18:11.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.498+0000 7fea779d7700 1 -- start start 2023-03-03T06:18:11.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.499+0000 7fea779d7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea700bc1f0 0x7fea7015f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.499+0000 7fea779d7700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea700b9310 con 0x7fea7015fbe0 2023-03-03T06:18:11.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.499+0000 7fea779d7700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea700bc6f0 con 0x7fea700c19d0 2023-03-03T06:18:11.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.499+0000 7fea779d7700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea700bc9d0 con 0x7fea700bc1f0 2023-03-03T06:18:11.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7fea74f72700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea700bc1f0 0x7fea7015f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7fea74f72700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea700bc1f0 0x7fea7015f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39050/0 (socket says 172.21.15.90:39050) 2023-03-03T06:18:11.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7fea74f72700 1 -- 172.21.15.90:0/3588390521 learned_addr learned my addr 172.21.15.90:0/3588390521 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7fb61c241700 1 Processor -- start 2023-03-03T06:18:11.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7f059bfff700 1 -- 172.21.15.90:0/283804299 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f05a4085580 con 0x7f05b40bcba0 2023-03-03T06:18:11.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7f05b27fc700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05a00682f0 0x7f05a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.500+0000 7fea74f72700 1 -- 172.21.15.90:0/3588390521 >> v1:172.21.15.121:6789/0 conn(0x7fea700c19d0 legacy=0x7fea700b8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fea74f72700 1 -- 172.21.15.90:0/3588390521 >> v1:172.21.15.90:6789/0 conn(0x7fea7015fbe0 legacy=0x7fea70166960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fea74f72700 1 -- 172.21.15.90:0/3588390521 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700b6920 con 0x7fea700bc1f0 2023-03-03T06:18:11.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fea74f72700 1 --2- 172.21.15.90:0/3588390521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea700bc1f0 0x7fea7015f6a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fea60009d80 tx=0x7fea6000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b02d4d961c096a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7f05b27fc700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05a00682f0 0x7f05a006a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f05b41e3620 tx=0x7f05a80090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fea67fff700 1 -- 172.21.15.90:0/3588390521 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6000e070 con 0x7fea700bc1f0 2023-03-03T06:18:11.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fea67fff700 1 -- 172.21.15.90:0/3588390521 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea600092e0 con 0x7fea700bc1f0 2023-03-03T06:18:11.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fea67fff700 1 -- 172.21.15.90:0/3588390521 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6000bb80 con 0x7fea700bc1f0 2023-03-03T06:18:11.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fa6a778f700 1 Processor -- start 2023-03-03T06:18:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7feddd3d2700 1 -- 172.21.15.90:0/3455950844 shutdown_connections 2023-03-03T06:18:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7f05b9a1d700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6824/96763,v1:172.21.15.90:6825/96763] conn(0x7f05b4072780 0x7f05b4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 --> [v2:172.21.15.90:6824/96763,v1:172.21.15.90:6825/96763] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05b414a730 con 0x7f05b4072780 2023-03-03T06:18:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fea779d7700 1 -- 172.21.15.90:0/3588390521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea700bc1f0 msgr2=0x7fea7015f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fea779d7700 1 --2- 172.21.15.90:0/3588390521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea700bc1f0 0x7fea7015f6a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fea60009d80 tx=0x7fea6000b330 comp rx=0 tx=0).stop 2023-03-03T06:18:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fea779d7700 1 -- 172.21.15.90:0/3588390521 shutdown_connections 2023-03-03T06:18:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fea779d7700 1 --2- 172.21.15.90:0/3588390521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea700bc1f0 0x7fea7015f6a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fea779d7700 1 -- 172.21.15.90:0/3588390521 >> 172.21.15.90:0/3588390521 conn(0x7fea7006afc0 msgr2=0x7fea700c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7f05b37fe700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6824/96763,v1:172.21.15.90:6825/96763] conn(0x7f05b4072780 0x7f05b4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7fa6a778f700 1 -- start start 2023-03-03T06:18:11.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fa6a778f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd1e0 0x7fa6a009d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fa6a778f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6a00bd630 0x7fa6a009a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fa6a778f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 0x7fa6a0071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fa6a778f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0070d30 con 0x7fa6a00bd630 2023-03-03T06:18:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fa6a778f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0070690 con 0x7fa6a00c19d0 2023-03-03T06:18:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7fa6a778f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a01f1320 con 0x7fa6a00bd1e0 2023-03-03T06:18:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a4d2a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6a00bd630 0x7fa6a009a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a552b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd1e0 0x7fa6a009d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a552b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd1e0 0x7fa6a009d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39060/0 (socket says 172.21.15.90:39060) 2023-03-03T06:18:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a552b700 1 -- 172.21.15.90:0/635062839 learned_addr learned my addr 172.21.15.90:0/635062839 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a5d2c700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 0x7fa6a0071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a4d2a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6a00bd630 0x7fa6a009a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34126/0 (socket says 172.21.15.90:34126) 2023-03-03T06:18:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7f05b37fe700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6824/96763,v1:172.21.15.90:6825/96763] conn(0x7f05b4072780 0x7f05b4070850 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a5d2c700 1 -- 172.21.15.90:0/635062839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd1e0 msgr2=0x7fa6a009d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a5d2c700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6a00bd1e0 0x7fa6a009d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a5d2c700 1 -- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6a00bd630 msgr2=0x7fa6a009a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a5d2c700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6a00bd630 0x7fa6a009a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a5d2c700 1 -- 172.21.15.90:0/635062839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6900094a0 con 0x7fa6a00c19d0 2023-03-03T06:18:11.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a4d2a700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6a00bd630 0x7fa6a009a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:11.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fea779d7700 1 -- 172.21.15.90:0/3588390521 shutdown_connections 2023-03-03T06:18:11.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fea779d7700 1 -- 172.21.15.90:0/3588390521 wait complete. 2023-03-03T06:18:11.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa6a5d2c700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 0x7fa6a0071150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa698009d90 tx=0x7fa69800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.503+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698003c20 con 0x7fa6a00c19d0 2023-03-03T06:18:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7feddd3d2700 1 -- 172.21.15.90:0/3455950844 wait complete. 2023-03-03T06:18:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7feddd3d2700 1 Processor -- start 2023-03-03T06:18:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7f059bfff700 1 -- 172.21.15.90:0/283804299 <== osd.3 v2:172.21.15.90:6824/96763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f05b414a730 con 0x7f05b4072780 2023-03-03T06:18:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7f7302359700 1 Processor -- start 2023-03-03T06:18:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7fea779d7700 1 Processor -- start 2023-03-03T06:18:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7fb61c241700 1 -- start start 2023-03-03T06:18:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fb61c241700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6140bb9d0 0x7fb614153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fb61c241700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6140b81f0 con 0x7fb6140bad50 2023-03-03T06:18:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fb61c241700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6140b7f10 con 0x7fb6140a6e80 2023-03-03T06:18:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fb61c241700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6140b6310 con 0x7fb6140bb9d0 2023-03-03T06:18:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7f05d57fa700 1 -- 172.21.15.90:0/741191565 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f05d0007cb0 con 0x7f05d80b5fd0 2023-03-03T06:18:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7f05d57fa700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05c40682f0 0x7f05c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7f05d7fff700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05c40682f0 0x7f05c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.501+0000 7f05d57fa700 1 -- 172.21.15.90:0/741191565 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f05d0087e60 con 0x7f05d80b5fd0 2023-03-03T06:18:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.502+0000 7f05d7fff700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05c40682f0 0x7f05c406a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f05d80baf60 tx=0x7f05c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7f7302359700 1 -- start start 2023-03-03T06:18:11.540 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f7302359700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f72fc145510 0x7f72fc14f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f7302359700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f72fc0bd3f0 0x7f72fc0bc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f7302359700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc0bc4a0 con 0x7f72fc0b5fd0 2023-03-03T06:18:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f7302359700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc0b7e40 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f7302359700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc14f6d0 con 0x7f72fc145510 2023-03-03T06:18:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fb61a7de700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb6140a6e80 0x7fb61415dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:45982/0 (socket says 172.21.15.90:45982) 2023-03-03T06:18:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fb61a7de700 1 -- 172.21.15.90:0/2288302630 learned_addr learned my addr 172.21.15.90:0/2288302630 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fb6197dc700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6140bb9d0 0x7fb614153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7feddd3d2700 1 -- start start 2023-03-03T06:18:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7feddd3d2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 0x7fedd809b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7feddd3d2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80bd6e0 0x7fedd8099e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7feddd3d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedd80c19d0 0x7fedd80721c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7feddd3d2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8071460 con 0x7fedd80a6e80 2023-03-03T06:18:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7feddd3d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80b9400 con 0x7fedd80c19d0 2023-03-03T06:18:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7feddd3d2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80b96e0 con 0x7fedd80bd6e0 2023-03-03T06:18:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fedd77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedd80c19d0 0x7fedd80721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fedd67fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80bd6e0 0x7fedd8099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fedd67fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80bd6e0 0x7fedd8099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39080/0 (socket says 172.21.15.90:39080) 2023-03-03T06:18:11.544 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fedd67fc700 1 -- 172.21.15.90:0/3703550559 learned_addr learned my addr 172.21.15.90:0/3703550559 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.544 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383522968 0 0) 0x7fb6140b81f0 con 0x7fb6140bad50 2023-03-03T06:18:11.544 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6000035b0 con 0x7fb6140bad50 2023-03-03T06:18:11.544 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fedd6ffd700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 0x7fedd809b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184336431 0 0) 0x7fb6140b7f10 con 0x7fb6140a6e80 2023-03-03T06:18:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6140b81f0 con 0x7fb6140a6e80 2023-03-03T06:18:11.546 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedd67fc700 1 -- 172.21.15.90:0/3703550559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedd80c19d0 msgr2=0x7fedd80721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.546 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedd67fc700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedd80c19d0 0x7fedd80721c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.546 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedd67fc700 1 -- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 msgr2=0x7fedd809b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.546 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedd67fc700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd80a6e80 0x7fedd809b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.546 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedd67fc700 1 -- 172.21.15.90:0/3703550559 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8009550 con 0x7fedd80bd6e0 2023-03-03T06:18:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69800f040 con 0x7fa6a00c19d0 2023-03-03T06:18:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800c880 con 0x7fa6a00c19d0 2023-03-03T06:18:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.504+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a00c1da0 con 0x7fa6a00c19d0 2023-03-03T06:18:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a01500e0 con 0x7fa6a00c19d0 2023-03-03T06:18:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6a0070510 con 0x7fa6a00c19d0 2023-03-03T06:18:11.548 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7fea779d7700 1 -- start start 2023-03-03T06:18:11.548 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea779d7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea700bc1f0 0x7fea700708c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.548 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea779d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700c19d0 0x7fea70159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.548 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea779d7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea7015fbe0 0x7fea701598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.548 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea779d7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700baf00 con 0x7fea700bc1f0 2023-03-03T06:18:11.549 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea779d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700b9ea0 con 0x7fea700c19d0 2023-03-03T06:18:11.549 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea779d7700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea700ba140 con 0x7fea7015fbe0 2023-03-03T06:18:11.549 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea74f72700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700c19d0 0x7fea70159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.549 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fea74f72700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700c19d0 0x7fea70159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48296/0 (socket says 172.21.15.90:48296) 2023-03-03T06:18:11.549 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea74f72700 1 -- 172.21.15.90:0/2455257840 learned_addr learned my addr 172.21.15.90:0/2455257840 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.549 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea75f74700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea7015fbe0 0x7fea701598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.550 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa69800c9e0 con 0x7fa6a00c19d0 2023-03-03T06:18:11.550 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea74f72700 1 -- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea7015fbe0 msgr2=0x7fea701598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.550 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea74f72700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea7015fbe0 0x7fea701598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.550 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea74f72700 1 -- 172.21.15.90:0/2455257840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea700bc1f0 msgr2=0x7fea700708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.551 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea74f72700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea700bc1f0 0x7fea700708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.551 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea74f72700 1 -- 172.21.15.90:0/2455257840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea60009a30 con 0x7fea700c19d0 2023-03-03T06:18:11.551 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea75f74700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea7015fbe0 0x7fea701598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:11.551 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fa695ffb700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa6840682c0 0x7fa68406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.552 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fea74f72700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700c19d0 0x7fea70159370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fea60004000 tx=0x7fea60004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.552 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa698086050 con 0x7fa6a00c19d0 2023-03-03T06:18:11.552 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fa695ffb700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6813/96747,v1:172.21.15.90:6815/96747] conn(0x7fa684070af0 0x7fa684072f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.553 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 --> [v2:172.21.15.90:6813/96747,v1:172.21.15.90:6815/96747] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6840735b0 con 0x7fa684070af0 2023-03-03T06:18:11.553 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa698088720 con 0x7fa6a00c19d0 2023-03-03T06:18:11.553 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fa6a4d2a700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6813/96747,v1:172.21.15.90:6815/96747] conn(0x7fa684070af0 0x7fa684072f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.553 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fa6a4d2a700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6813/96747,v1:172.21.15.90:6815/96747] conn(0x7fa684070af0 0x7fa684072f00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.553 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== osd.0 v2:172.21.15.90:6813/96747 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa6840735b0 con 0x7fa684070af0 2023-03-03T06:18:11.553 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f72fbfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f72fc145510 0x7f72fc14f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f72fbfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f72fc145510 0x7f72fc14f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48276/0 (socket says 172.21.15.90:48276) 2023-03-03T06:18:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f72fbfff700 1 -- 172.21.15.90:0/3123152443 learned_addr learned my addr 172.21.15.90:0/3123152443 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.505+0000 7f73008f6700 1 --2- 172.21.15.90:0/3123152443 >> v2:172.21.15.90:3300/0 conn(0x7f72fc0bd3f0 0x7f72fc0bc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72fbfff700 1 -- 172.21.15.90:0/3123152443 >> v2:172.21.15.90:3300/0 conn(0x7f72fc0bd3f0 msgr2=0x7f72fc0bc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72fbfff700 1 --2- 172.21.15.90:0/3123152443 >> v2:172.21.15.90:3300/0 conn(0x7f72fc0bd3f0 0x7f72fc0bc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72fbfff700 1 -- 172.21.15.90:0/3123152443 >> v1:172.21.15.121:6789/0 conn(0x7f72fc0b5fd0 legacy=0x7f72fc06c8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T06:18:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72fbfff700 1 -- 172.21.15.90:0/3123152443 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc1458e0 con 0x7f72fc145510 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72fbfff700 1 --2- 172.21.15.90:0/3123152443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 0x7f72fc14f0f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72fc06c2b0 tx=0x7f72ec009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9d866cd75362f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72faffd700 1 -- 172.21.15.90:0/3123152443 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec004030 con 0x7f72fc145510 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72faffd700 1 -- 172.21.15.90:0/3123152443 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec004190 con 0x7f72fc145510 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7f72faffd700 1 -- 172.21.15.90:0/3123152443 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec004520 con 0x7f72fc145510 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7f7302359700 1 -- 172.21.15.90:0/3123152443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 msgr2=0x7f72fc14f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7f7302359700 1 --2- 172.21.15.90:0/3123152443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 0x7f72fc14f0f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72fc06c2b0 tx=0x7f72ec009520 comp rx=0 tx=0).stop 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7f7302359700 1 -- 172.21.15.90:0/3123152443 shutdown_connections 2023-03-03T06:18:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7f7302359700 1 --2- 172.21.15.90:0/3123152443 >> v2:172.21.15.90:3300/0 conn(0x7f72fc0bd3f0 0x7f72fc0bc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.556 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7f7302359700 1 --2- 172.21.15.90:0/3123152443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 0x7f72fc14f0f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.556 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7f7302359700 1 -- 172.21.15.90:0/3123152443 >> 172.21.15.90:0/3123152443 conn(0x7f72fc06afc0 msgr2=0x7f72fc0c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.556 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7f7302359700 1 -- 172.21.15.90:0/3123152443 shutdown_connections 2023-03-03T06:18:11.556 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7f7302359700 1 -- 172.21.15.90:0/3123152443 wait complete. 2023-03-03T06:18:11.556 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea600038a0 con 0x7fea700c19d0 2023-03-03T06:18:11.557 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea60003a00 con 0x7fea700c19d0 2023-03-03T06:18:11.557 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fea779d7700 1 -- 172.21.15.90:0/2455257840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea700b6ee0 con 0x7fea700c19d0 2023-03-03T06:18:11.557 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60003db0 con 0x7fea700c19d0 2023-03-03T06:18:11.557 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fea779d7700 1 -- 172.21.15.90:0/2455257840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea700b6c00 con 0x7fea700c19d0 2023-03-03T06:18:11.557 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedd67fc700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80bd6e0 0x7fedd8099e20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fedd0001ed0 tx=0x7fedd000a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.557 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd000ade0 con 0x7fedd80bd6e0 2023-03-03T06:18:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd00020d0 con 0x7fedd80bd6e0 2023-03-03T06:18:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0003c20 con 0x7fedd80bd6e0 2023-03-03T06:18:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8153fa0 con 0x7fedd80bd6e0 2023-03-03T06:18:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd806cde0 con 0x7fedd80bd6e0 2023-03-03T06:18:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fedd80bb130 con 0x7fedd80bd6e0 2023-03-03T06:18:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fedd0010050 con 0x7fedd80bd6e0 2023-03-03T06:18:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.510+0000 7fedbf7fe700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fedc4068340 0x7fedc406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.510+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fedd0085e80 con 0x7fedd80bd6e0 2023-03-03T06:18:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.510+0000 7fedbf7fe700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6800/96746,v1:172.21.15.90:6801/96746] conn(0x7fedc4070bf0 0x7fedc4073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.510+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 --> [v2:172.21.15.90:6800/96746,v1:172.21.15.90:6801/96746] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fedc40736b0 con 0x7fedc4070bf0 2023-03-03T06:18:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.510+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedd0086300 con 0x7fedd80bd6e0 2023-03-03T06:18:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.510+0000 7fedd77fe700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6800/96746,v1:172.21.15.90:6801/96746] conn(0x7fedc4070bf0 0x7fedc4073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.559 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.511+0000 7fedd77fe700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6800/96746,v1:172.21.15.90:6801/96746] conn(0x7fedc4070bf0 0x7fedc4073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.511+0000 7fedd6ffd700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fedc4068340 0x7fedc406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235986059 0 0) 0x7fb6140b81f0 con 0x7fb6140a6e80 2023-03-03T06:18:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6140bb9d0 msgr2=0x7fb614153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6140bb9d0 0x7fb614153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 >> v1:172.21.15.90:6789/0 conn(0x7fb6140bad50 legacy=0x7fb6140c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6140b65f0 con 0x7fb6140a6e80 2023-03-03T06:18:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7fb610001670 con 0x7fb6140a6e80 2023-03-03T06:18:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 >> v1:172.21.15.121:6789/0 conn(0x7fb6140a6e80 legacy=0x7fb61415dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb600003c30 0x7fb600004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600004850 0x7fb600004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb600005250 0x7fb60000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604001160 con 0x7fb600004850 2023-03-03T06:18:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6140b6310 con 0x7fb600005250 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.506+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60000c1e0 con 0x7fb600003c30 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fb61a7de700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb600005250 0x7fb60000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.507+0000 7fb619fdd700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb600003c30 0x7fb600004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb619fdd700 1 -- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb600005250 msgr2=0x7fb60000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb619fdd700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb600005250 0x7fb60000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb619fdd700 1 -- 172.21.15.90:0/2288302630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600004850 msgr2=0x7fb600004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb619fdd700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600004850 0x7fb600004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.562 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb619fdd700 1 -- 172.21.15.90:0/2288302630 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6140b65f0 con 0x7fb600003c30 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb619fdd700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb600003c30 0x7fb600004000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb604007ec0 tx=0x7fb600005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a52f594f7a66361c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6040052d0 con 0x7fb600003c30 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb604005450 con 0x7fb600003c30 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.508+0000 7fb618fdb700 1 -- 172.21.15.90:0/2288302630 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60400cbf0 con 0x7fb600003c30 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fb61c241700 1 -- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb600003c30 msgr2=0x7fb600004000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fb61c241700 1 --2- 172.21.15.90:0/2288302630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb600003c30 0x7fb600004000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb604007ec0 tx=0x7fb600005880 comp rx=0 tx=0).stop 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fb61a7de700 1 -- 172.21.15.90:0/2288302630 reap_dead start 2023-03-03T06:18:11.563 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fb61c241700 1 -- 172.21.15.90:0/2288302630 shutdown_connections 2023-03-03T06:18:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.509+0000 7fb61c241700 1 -- 172.21.15.90:0/2288302630 >> 172.21.15.90:0/2288302630 conn(0x7fb61406afc0 msgr2=0x7fb6140c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.511+0000 7f7302359700 1 Processor -- start 2023-03-03T06:18:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.511+0000 7f05deee1700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:6800/96841,v1:172.21.15.121:6801/96841] conn(0x7f05d814b130 0x7f05d8143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.511+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 --> [v2:172.21.15.121:6800/96841,v1:172.21.15.121:6801/96841] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05d81ecc30 con 0x7f05d814b130 2023-03-03T06:18:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.511+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== osd.1 v2:172.21.15.90:6800/96746 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fedc40736b0 con 0x7fedc4070bf0 2023-03-03T06:18:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f05dd47e700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:6800/96841,v1:172.21.15.121:6801/96841] conn(0x7f05d814b130 0x7f05d8143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.565 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fea700b8660 con 0x7fea700c19d0 2023-03-03T06:18:11.566 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7fedd6ffd700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fedc4068340 0x7fedc406a7f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009870 tx=0x7fedcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.511+0000 7f7302359700 1 -- start start 2023-03-03T06:18:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f7302359700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72fc0bd3f0 0x7f72fc149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f7302359700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 0x7f72fc146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.567 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f7302359700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72fc147000 0x7f72fc147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f7302359700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc13bc30 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f7302359700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc13fcc0 con 0x7f72fc145510 2023-03-03T06:18:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f7302359700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc13f830 con 0x7f72fc147000 2023-03-03T06:18:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f72fbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72fc0bd3f0 0x7f72fc149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f72fbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72fc0bd3f0 0x7f72fc149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34170/0 (socket says 172.21.15.90:34170) 2023-03-03T06:18:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f72fbfff700 1 -- 172.21.15.90:0/3630115340 learned_addr learned my addr 172.21.15.90:0/3630115340 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f72fb7fe700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 0x7f72fc146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.512+0000 7f73008f6700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72fc147000 0x7f72fc147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72fbfff700 1 -- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72fc147000 msgr2=0x7f72fc147480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72fbfff700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72fc147000 0x7f72fc147480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72fbfff700 1 -- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 msgr2=0x7f72fc146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72fbfff700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72fc145510 0x7f72fc146480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72fbfff700 1 -- 172.21.15.90:0/3630115340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec00a5f0 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72fbfff700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72fc0bd3f0 0x7f72fc149640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009920 tx=0x7f72ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec0038a0 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec003a00 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec003d90 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc0bbe50 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.513+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc0bbb70 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.514+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72fc0bc130 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.515+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72ec028030 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f05dd47e700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:6800/96841,v1:172.21.15.121:6801/96841] conn(0x7f05d814b130 0x7f05d8143310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f72f8ff9700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f72e80682f0 0x7f72e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f72ec087620 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f72fb7fe700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f72e80682f0 0x7f72e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f72f8ff9700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:6808/96842,v1:172.21.15.121:6809/96842] conn(0x7f72e8070b20 0x7f72e8072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 --> [v2:172.21.15.121:6808/96842,v1:172.21.15.121:6809/96842] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72e80735e0 con 0x7f72e8070b20 2023-03-03T06:18:11.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72ec087a80 con 0x7f72fc0bd3f0 2023-03-03T06:18:11.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.516+0000 7f72fb7fe700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f72e80682f0 0x7f72e806a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f72fc1e4b60 tx=0x7f72f0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fea6002c020 con 0x7fea700c19d0 2023-03-03T06:18:11.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7f05d57fa700 1 -- 172.21.15.90:0/741191565 <== osd.5 v2:172.21.15.121:6800/96841 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f05d81ecc30 con 0x7f05d814b130 2023-03-03T06:18:11.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fea65ffb700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fea540683c0 0x7fea5406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.573 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fa6a552b700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa6840682c0 0x7fa68406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fea60086870 con 0x7fea700c19d0 2023-03-03T06:18:11.574 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fa6a552b700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa6840682c0 0x7fa68406a770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa690009b60 tx=0x7fa69001c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fea75773700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fea540683c0 0x7fea5406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fea65ffb700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:6824/96852,v1:172.21.15.121:6825/96852] conn(0x7fea54070c70 0x7fea54073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 --> [v2:172.21.15.121:6824/96852,v1:172.21.15.121:6825/96852] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea54073730 con 0x7fea54070c70 2023-03-03T06:18:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.517+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea6008b2a0 con 0x7fea700c19d0 2023-03-03T06:18:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.518+0000 7fea75f74700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:6824/96852,v1:172.21.15.121:6825/96852] conn(0x7fea54070c70 0x7fea54073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.518+0000 7f73008f6700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:6808/96842,v1:172.21.15.121:6809/96842] conn(0x7f72e8070b20 0x7f72e8072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.518+0000 7fea75773700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fea540683c0 0x7fea5406a870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea6c000f60 tx=0x7fea6c007a40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.518+0000 7fea75f74700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:6824/96852,v1:172.21.15.121:6825/96852] conn(0x7fea54070c70 0x7fea54073080 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.518+0000 7f73008f6700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:6808/96842,v1:172.21.15.121:6809/96842] conn(0x7f72e8070b20 0x7f72e8072f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.519+0000 7fb61c241700 1 -- 172.21.15.90:0/2288302630 shutdown_connections 2023-03-03T06:18:11.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.519+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== osd.7 v2:172.21.15.121:6824/96852 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fea54073730 con 0x7fea54070c70 2023-03-03T06:18:11.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.519+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== osd.6 v2:172.21.15.121:6808/96842 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f72e80735e0 con 0x7f72e8070b20 2023-03-03T06:18:11.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.529+0000 7fb61c241700 1 -- 172.21.15.90:0/2288302630 wait complete. 2023-03-03T06:18:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.530+0000 7fb61c241700 1 Processor -- start 2023-03-03T06:18:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.530+0000 7fb61c241700 1 -- start start 2023-03-03T06:18:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.531+0000 7fb61c241700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600003c30 0x7fb614151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.531+0000 7fb61c241700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb614149ab0 0x7fb61414ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.531+0000 7fb61c241700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb614147810 0x7fb614147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.531+0000 7fb61c241700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61406f4c0 con 0x7fb600003c30 2023-03-03T06:18:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.531+0000 7fb61c241700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61406f9b0 con 0x7fb614147810 2023-03-03T06:18:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.531+0000 7fb61c241700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6141481c0 con 0x7fb614149ab0 2023-03-03T06:18:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb614149ab0 0x7fb61414ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb61a7de700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb614147810 0x7fb614147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb614149ab0 0x7fb61414ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39120/0 (socket says 172.21.15.90:39120) 2023-03-03T06:18:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 -- 172.21.15.90:0/1046382828 learned_addr learned my addr 172.21.15.90:0/1046382828 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 -- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb614147810 msgr2=0x7fb614147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb614147810 0x7fb614147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 -- 172.21.15.90:0/1046382828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600003c30 msgr2=0x7fb614151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb600003c30 0x7fb614151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 -- 172.21.15.90:0/1046382828 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6140c1e70 con 0x7fb614149ab0 2023-03-03T06:18:11.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.532+0000 7fb6197dc700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb614149ab0 0x7fb61414ef70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00ef70 tx=0x7fb60c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 --> [v2:172.21.15.90:6813/96747,v1:172.21.15.90:6815/96747] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6a00707d0 con 0x7fa684070af0 2023-03-03T06:18:11.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 --> [v2:172.21.15.90:6824/96763,v1:172.21.15.90:6825/96763] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f05b40bc980 con 0x7f05b4072780 2023-03-03T06:18:11.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c00cc60 con 0x7fb614149ab0 2023-03-03T06:18:11.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60c00f040 con 0x7fb614149ab0 2023-03-03T06:18:11.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c009e70 con 0x7fb614149ab0 2023-03-03T06:18:11.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7f87a7e71700 1 Processor -- start 2023-03-03T06:18:11.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7f059bfff700 1 -- 172.21.15.90:0/283804299 <== osd.3 v2:172.21.15.90:6824/96763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f05b40bc980 con 0x7f05b4072780 2023-03-03T06:18:11.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6140b68d0 con 0x7fb614149ab0 2023-03-03T06:18:11.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.536+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6140bb120 con 0x7fb614149ab0 2023-03-03T06:18:11.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.537+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 --> [v2:172.21.15.121:6808/96842,v1:172.21.15.121:6809/96842] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f72fc13fb10 con 0x7f72e8070b20 2023-03-03T06:18:11.584 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.537+0000 7f87a7e71700 1 -- start start 2023-03-03T06:18:11.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a7e71700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f87a00bc970 0x7f87a0153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a7e71700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bd9a0 0x7f87a0153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a7e71700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a00b6350 con 0x7f87a00c1e30 2023-03-03T06:18:11.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a7e71700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a00b6ed0 con 0x7f87a00bc970 2023-03-03T06:18:11.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a7e71700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00b6630 con 0x7f87a00bd9a0 2023-03-03T06:18:11.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb60c014480 con 0x7fb614149ab0 2023-03-03T06:18:11.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f72f8ff9700 1 -- 172.21.15.90:0/3630115340 <== osd.6 v2:172.21.15.121:6808/96842 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f72fc13fb10 con 0x7f72e8070b20 2023-03-03T06:18:11.586 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7fa695ffb700 1 -- 172.21.15.90:0/635062839 <== osd.0 v2:172.21.15.90:6813/96747 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa69c009de0 con 0x7fa684070af0 2023-03-03T06:18:11.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a5c0d700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f87a00c1e30 0x7f87a014d550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:46038/0 (socket says 172.21.15.90:46038) 2023-03-03T06:18:11.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a5c0d700 1 -- 172.21.15.90:0/2706999245 learned_addr learned my addr 172.21.15.90:0/2706999245 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:11.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a640e700 1 --2- 172.21.15.90:0/2706999245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bd9a0 0x7f87a0153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a540c700 1 --2- 172.21.15.90:0/2706999245 >> v2:172.21.15.121:3300/0 conn(0x7f87a00bc970 0x7f87a0153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5fc000fb0 con 0x7fb614149ab0 2023-03-03T06:18:11.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7fb60a7fc700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb60008bd30 0x7fb60008e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:6808/96842,v1:172.21.15.121:6809/96842] conn(0x7f72e8070b20 msgr2=0x7f72e8072f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f7302359700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.121:6808/96842,v1:172.21.15.121:6809/96842] conn(0x7f72e8070b20 0x7f72e8072f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a4c0b700 1 -- 172.21.15.90:0/2706999245 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458785711 0 0) 0x7f87a00b6350 con 0x7f87a00c1e30 2023-03-03T06:18:11.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a4c0b700 1 -- 172.21.15.90:0/2706999245 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878c0035b0 con 0x7f87a00c1e30 2023-03-03T06:18:11.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a640e700 1 -- 172.21.15.90:0/2706999245 >> v2:172.21.15.121:3300/0 conn(0x7f87a00bc970 msgr2=0x7f87a0153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a640e700 1 --2- 172.21.15.90:0/2706999245 >> v2:172.21.15.121:3300/0 conn(0x7f87a00bc970 0x7f87a0153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a640e700 1 -- 172.21.15.90:0/2706999245 >> v1:172.21.15.121:6789/0 conn(0x7f87a00c1e30 legacy=0x7f87a014d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.538+0000 7f87a640e700 1 -- 172.21.15.90:0/2706999245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a00b6910 con 0x7f87a00bd9a0 2023-03-03T06:18:11.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f87a640e700 1 --2- 172.21.15.90:0/2706999245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bd9a0 0x7f87a0153fe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8798009d90 tx=0x7f879800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f53291dcc1460e88 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f72e80682f0 msgr2=0x7f72e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f72e80682f0 0x7f72e806a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f72fc1e4b60 tx=0x7f72f0009190 comp rx=0 tx=0).stop 2023-03-03T06:18:11.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72fc0bd3f0 msgr2=0x7f72fc149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 --2- 172.21.15.90:0/3630115340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72fc0bd3f0 0x7f72fc149640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009920 tx=0x7f72ec004750 comp rx=0 tx=0).stop 2023-03-03T06:18:11.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f87a4c0b700 1 -- 172.21.15.90:0/2706999245 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800e070 con 0x7f87a00bd9a0 2023-03-03T06:18:11.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f87a4c0b700 1 -- 172.21.15.90:0/2706999245 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879800bae0 con 0x7f87a00bd9a0 2023-03-03T06:18:11.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f87a4c0b700 1 -- 172.21.15.90:0/2706999245 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800be70 con 0x7f87a00bd9a0 2023-03-03T06:18:11.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb60c0868e0 con 0x7fb614149ab0 2023-03-03T06:18:11.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7fb60a7fc700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:6816/96843,v1:172.21.15.121:6817/96843] conn(0x7fb6000944e0 0x7fb6000968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 --> [v2:172.21.15.121:6816/96843,v1:172.21.15.121:6817/96843] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb600096fa0 con 0x7fb6000944e0 2023-03-03T06:18:11.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb60c053070 con 0x7fb614149ab0 2023-03-03T06:18:11.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7fb619fdd700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb60008bd30 0x7fb60008e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7fb61a7de700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:6816/96843,v1:172.21.15.121:6817/96843] conn(0x7fb6000944e0 0x7fb6000968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f73008f6700 1 -- 172.21.15.90:0/3630115340 reap_dead start 2023-03-03T06:18:11.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 shutdown_connections 2023-03-03T06:18:11.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 >> 172.21.15.90:0/3630115340 conn(0x7f72fc06afc0 msgr2=0x7f72fc0ba5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 shutdown_connections 2023-03-03T06:18:11.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.539+0000 7f7302359700 1 -- 172.21.15.90:0/3630115340 wait complete. 2023-03-03T06:18:11.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6824/96763,v1:172.21.15.90:6825/96763] conn(0x7f05b4072780 msgr2=0x7f05b4070850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6824/96763,v1:172.21.15.90:6825/96763] conn(0x7f05b4072780 0x7f05b4070850 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05a00682f0 msgr2=0x7f05a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05a00682f0 0x7f05a006a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f05b41e3620 tx=0x7f05a80090e0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b40bcba0 msgr2=0x7f05b4071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 --2- 172.21.15.90:0/283804299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05b40bcba0 0x7f05b4071d30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f05a400dbd0 tx=0x7f05a400dee0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b37fe700 1 -- 172.21.15.90:0/283804299 reap_dead start 2023-03-03T06:18:11.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 -- 172.21.15.90:0/2706999245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bd9a0 msgr2=0x7f87a0153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 --2- 172.21.15.90:0/2706999245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bd9a0 0x7f87a0153fe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8798009d90 tx=0x7f879800b330 comp rx=0 tx=0).stop 2023-03-03T06:18:11.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fb619fdd700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb60008bd30 0x7fb60008e1e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb604007ec0 tx=0x7fb60400a000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 shutdown_connections 2023-03-03T06:18:11.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 >> 172.21.15.90:0/283804299 conn(0x7f05b406afc0 msgr2=0x7f05b40be070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 -- 172.21.15.90:0/2706999245 shutdown_connections 2023-03-03T06:18:11.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 --2- 172.21.15.90:0/2706999245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bd9a0 0x7f87a0153fe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6813/96747,v1:172.21.15.90:6815/96747] conn(0x7fa684070af0 msgr2=0x7fa684072f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 --2- 172.21.15.90:0/2706999245 >> v2:172.21.15.121:3300/0 conn(0x7f87a00bc970 0x7f87a0153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.597 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 -- 172.21.15.90:0/2706999245 >> 172.21.15.90:0/2706999245 conn(0x7f87a006c6c0 msgr2=0x7f87a00c14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fa6a778f700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6813/96747,v1:172.21.15.90:6815/96747] conn(0x7fa684070af0 0x7fa684072f00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 -- 172.21.15.90:0/2706999245 shutdown_connections 2023-03-03T06:18:11.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7f87a7e71700 1 -- 172.21.15.90:0/2706999245 wait complete. 2023-03-03T06:18:11.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fb61a7de700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:6816/96843,v1:172.21.15.121:6817/96843] conn(0x7fb6000944e0 0x7fb6000968f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa6840682c0 msgr2=0x7fa68406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fa6a778f700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa6840682c0 0x7fa68406a770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa690009b60 tx=0x7fa69001c040 comp rx=0 tx=0).stop 2023-03-03T06:18:11.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 msgr2=0x7fa6a0071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.599 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7fa6a778f700 1 --2- 172.21.15.90:0/635062839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6a00c19d0 0x7fa6a0071150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa698009d90 tx=0x7fa69800c3a0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.540+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 --> [v2:172.21.15.90:6800/96746,v1:172.21.15.90:6801/96746] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fedd8099aa0 con 0x7fedc4070bf0 2023-03-03T06:18:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.541+0000 7fa6a5d2c700 1 -- 172.21.15.90:0/635062839 reap_dead start 2023-03-03T06:18:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.541+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 shutdown_connections 2023-03-03T06:18:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.541+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 >> 172.21.15.90:0/635062839 conn(0x7fa6a006afc0 msgr2=0x7fa6a00ba9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.541+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== osd.4 v2:172.21.15.121:6816/96843 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb600096fa0 con 0x7fb6000944e0 2023-03-03T06:18:11.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.541+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 shutdown_connections 2023-03-03T06:18:11.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.541+0000 7fedbf7fe700 1 -- 172.21.15.90:0/3703550559 <== osd.1 v2:172.21.15.90:6800/96746 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fedd8099aa0 con 0x7fedc4070bf0 2023-03-03T06:18:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6800/96746,v1:172.21.15.90:6801/96746] conn(0x7fedc4070bf0 msgr2=0x7fedc4073000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6800/96746,v1:172.21.15.90:6801/96746] conn(0x7fedc4070bf0 0x7fedc4073000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fedc4068340 msgr2=0x7fedc406a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fedc4068340 0x7fedc406a7f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009870 tx=0x7fedcc008040 comp rx=0 tx=0).stop 2023-03-03T06:18:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80bd6e0 msgr2=0x7fedd8099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 --2- 172.21.15.90:0/3703550559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedd80bd6e0 0x7fedd8099e20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fedd0001ed0 tx=0x7fedd000a340 comp rx=0 tx=0).stop 2023-03-03T06:18:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7fedd77fe700 1 -- 172.21.15.90:0/3703550559 reap_dead start 2023-03-03T06:18:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 shutdown_connections 2023-03-03T06:18:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 >> 172.21.15.90:0/3703550559 conn(0x7fedd806afc0 msgr2=0x7fedd80c07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 shutdown_connections 2023-03-03T06:18:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7feddd3d2700 1 -- 172.21.15.90:0/3703550559 wait complete. 2023-03-03T06:18:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.542+0000 7f87a7e71700 1 Processor -- start 2023-03-03T06:18:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.542+0000 7f87a7e71700 1 -- start start 2023-03-03T06:18:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.542+0000 7f87a7e71700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bc970 0x7f87a00a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.542+0000 7f87a7e71700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a00bd9a0 0x7f87a01df100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a7e71700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a00c1e30 0x7f87a00a6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a7e71700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a0072750 con 0x7f87a00bd9a0 2023-03-03T06:18:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a7e71700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a00a5640 con 0x7f87a00c1e30 2023-03-03T06:18:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a7e71700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a01e0cd0 con 0x7f87a00bc970 2023-03-03T06:18:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a00bd9a0 0x7f87a01df100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a00bd9a0 0x7f87a01df100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34200/0 (socket says 172.21.15.90:34200) 2023-03-03T06:18:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 -- 172.21.15.90:0/405286097 learned_addr learned my addr 172.21.15.90:0/405286097 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a640e700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a00c1e30 0x7f87a00a6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 -- 172.21.15.90:0/405286097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bc970 msgr2=0x7f87a00a5100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a00bc970 0x7f87a00a5100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 -- 172.21.15.90:0/405286097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a00c1e30 msgr2=0x7f87a00a6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a00c1e30 0x7f87a00a6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 -- 172.21.15.90:0/405286097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798009a70 con 0x7f87a00bd9a0 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a640e700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a00c1e30 0x7f87a00a6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a540c700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a00bd9a0 0x7f87a01df100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f879c00e840 tx=0x7f879c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87967fc700 1 -- 172.21.15.90:0/405286097 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c013610 con 0x7f87a00bd9a0 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a00c2200 con 0x7f87a00bd9a0 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87967fc700 1 -- 172.21.15.90:0/405286097 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879c013770 con 0x7f87a00bd9a0 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.543+0000 7f87967fc700 1 -- 172.21.15.90:0/405286097 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c013b20 con 0x7f87a00bd9a0 2023-03-03T06:18:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.544+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a00b6bf0 con 0x7f87a00bd9a0 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7f87967fc700 1 -- 172.21.15.90:0/405286097 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f879c023070 con 0x7f87a00bd9a0 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7f87967fc700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f878c068780 0x7f878c06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.545+0000 7f87967fc700 1 -- 172.21.15.90:0/405286097 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f879c085930 con 0x7f87a00bd9a0 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.546+0000 7f87a5c0d700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f878c068780 0x7f878c06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.546+0000 7f87a5c0d700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f878c068780 0x7f878c06ac30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f8790007a50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.547+0000 7f87a7e71700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6808/96756,v1:172.21.15.90:6809/96756] conn(0x7f87a014b6c0 0x7f87a01dfe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.547+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 --> [v2:172.21.15.90:6808/96756,v1:172.21.15.90:6809/96756] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a01def50 con 0x7f87a014b6c0 2023-03-03T06:18:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.548+0000 7f87a640e700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6808/96756,v1:172.21.15.90:6809/96756] conn(0x7f87a014b6c0 0x7f87a01dfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.548+0000 7f87a640e700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6808/96756,v1:172.21.15.90:6809/96756] conn(0x7f87a014b6c0 0x7f87a01dfe40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7f87967fc700 1 -- 172.21.15.90:0/405286097 <== osd.2 v2:172.21.15.90:6808/96756 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f87a01def50 con 0x7f87a014b6c0 2023-03-03T06:18:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.542+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 shutdown_connections 2023-03-03T06:18:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.548+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 --> [v2:172.21.15.121:6824/96852,v1:172.21.15.121:6825/96852] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fea7004e750 con 0x7fea54070c70 2023-03-03T06:18:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7fea65ffb700 1 -- 172.21.15.90:0/2455257840 <== osd.7 v2:172.21.15.121:6824/96852 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fea7004e750 con 0x7fea54070c70 2023-03-03T06:18:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 --> [v2:172.21.15.121:6800/96841,v1:172.21.15.121:6801/96841] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f05d804e750 con 0x7f05d814b130 2023-03-03T06:18:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7f05d57fa700 1 -- 172.21.15.90:0/741191565 <== osd.5 v2:172.21.15.121:6800/96841 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f05d804e750 con 0x7f05d814b130 2023-03-03T06:18:11.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:6824/96852,v1:172.21.15.121:6825/96852] conn(0x7fea54070c70 msgr2=0x7fea54073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.549+0000 7fea5b7fe700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:6824/96852,v1:172.21.15.121:6825/96852] conn(0x7fea54070c70 0x7fea54073080 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fea540683c0 msgr2=0x7fea5406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7fea5b7fe700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fea540683c0 0x7fea5406a870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea6c000f60 tx=0x7fea6c007a40 comp rx=0 tx=0).stop 2023-03-03T06:18:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700c19d0 msgr2=0x7fea70159370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7fea5b7fe700 1 --2- 172.21.15.90:0/2455257840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700c19d0 0x7fea70159370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fea60004000 tx=0x7fea60004750 comp rx=0 tx=0).stop 2023-03-03T06:18:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7fea75f74700 1 -- 172.21.15.90:0/2455257840 reap_dead start 2023-03-03T06:18:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 shutdown_connections 2023-03-03T06:18:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 >> 172.21.15.90:0/2455257840 conn(0x7fea7006afc0 msgr2=0x7fea70089980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:6800/96841,v1:172.21.15.121:6801/96841] conn(0x7f05d814b130 msgr2=0x7f05d8143310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:6800/96841,v1:172.21.15.121:6801/96841] conn(0x7f05d814b130 0x7f05d8143310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05c40682f0 msgr2=0x7f05c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f05c40682f0 0x7f05c406a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f05d80baf60 tx=0x7f05c8009040 comp rx=0 tx=0).stop 2023-03-03T06:18:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d80b5fd0 msgr2=0x7f05d806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 --2- 172.21.15.90:0/741191565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d80b5fd0 0x7f05d806f730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f05d00097a0 tx=0x7f05d0004ad0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05dd47e700 1 -- 172.21.15.90:0/741191565 reap_dead start 2023-03-03T06:18:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 shutdown_connections 2023-03-03T06:18:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 >> 172.21.15.90:0/741191565 conn(0x7f05d806afc0 msgr2=0x7f05d806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 shutdown_connections 2023-03-03T06:18:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.550+0000 7f05deee1700 1 -- 172.21.15.90:0/741191565 wait complete. 2023-03-03T06:18:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.552+0000 7fa6a778f700 1 -- 172.21.15.90:0/635062839 wait complete. 2023-03-03T06:18:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.552+0000 7f05b9a1d700 1 -- 172.21.15.90:0/283804299 wait complete. 2023-03-03T06:18:11.615 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:11.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-03-03T06:18:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.555+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 shutdown_connections 2023-03-03T06:18:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.555+0000 7fea5b7fe700 1 -- 172.21.15.90:0/2455257840 wait complete. 2023-03-03T06:18:11.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.568+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 --> [v2:172.21.15.90:6808/96756,v1:172.21.15.90:6809/96756] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8788000f80 con 0x7f87a014b6c0 2023-03-03T06:18:11.617 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87967fc700 1 -- 172.21.15.90:0/405286097 <== osd.2 v2:172.21.15.90:6808/96756 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8788000f80 con 0x7f87a014b6c0 2023-03-03T06:18:11.618 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:11.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-03T06:18:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6808/96756,v1:172.21.15.90:6809/96756] conn(0x7f87a014b6c0 msgr2=0x7f87a01dfe40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6808/96756,v1:172.21.15.90:6809/96756] conn(0x7f87a014b6c0 0x7f87a01dfe40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f878c068780 msgr2=0x7f878c06ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f878c068780 0x7f878c06ac30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c00 tx=0x7f8790007a50 comp rx=0 tx=0).stop 2023-03-03T06:18:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a00bd9a0 msgr2=0x7f87a01df100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 --2- 172.21.15.90:0/405286097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a00bd9a0 0x7f87a01df100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f879c00e840 tx=0x7f879c00eb50 comp rx=0 tx=0).stop 2023-03-03T06:18:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a640e700 1 -- 172.21.15.90:0/405286097 reap_dead start 2023-03-03T06:18:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 shutdown_connections 2023-03-03T06:18:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.569+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 >> 172.21.15.90:0/405286097 conn(0x7f87a006c6c0 msgr2=0x7f87a00c0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.570+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 shutdown_connections 2023-03-03T06:18:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.570+0000 7f87a7e71700 1 -- 172.21.15.90:0/405286097 wait complete. 2023-03-03T06:18:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.571+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 --> [v2:172.21.15.121:6816/96843,v1:172.21.15.121:6817/96843] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb5fc002d30 con 0x7fb6000944e0 2023-03-03T06:18:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb60a7fc700 1 -- 172.21.15.90:0/1046382828 <== osd.4 v2:172.21.15.121:6816/96843 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb5fc002d30 con 0x7fb6000944e0 2023-03-03T06:18:11.622 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:11.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-03T06:18:11.623 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:6816/96843,v1:172.21.15.121:6817/96843] conn(0x7fb6000944e0 msgr2=0x7fb6000968f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.623 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:6816/96843,v1:172.21.15.121:6817/96843] conn(0x7fb6000944e0 0x7fb6000968f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.624 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb60008bd30 msgr2=0x7fb60008e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.624 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fb60008bd30 0x7fb60008e1e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb604007ec0 tx=0x7fb60400a000 comp rx=0 tx=0).stop 2023-03-03T06:18:11.624 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb614149ab0 msgr2=0x7fb61414ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.624 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 --2- 172.21.15.90:0/1046382828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb614149ab0 0x7fb61414ef70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00ef70 tx=0x7fb60c00c3a0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.624 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61a7de700 1 -- 172.21.15.90:0/1046382828 reap_dead start 2023-03-03T06:18:11.625 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:11.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-03-03T06:18:11.626 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 shutdown_connections 2023-03-03T06:18:11.626 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.572+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 >> 172.21.15.90:0/1046382828 conn(0x7fb61406afc0 msgr2=0x7fb6140bc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.573+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 shutdown_connections 2023-03-03T06:18:11.627 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.573+0000 7fb61c241700 1 -- 172.21.15.90:0/1046382828 wait complete. 2023-03-03T06:18:11.628 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:11.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-03-03T06:18:11.629 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:11.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-03-03T06:18:11.630 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:11.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-03-03T06:18:11.631 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:11.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-03T06:18:11.866 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.865+0000 7f6f5ae1f700 1 Processor -- start 2023-03-03T06:18:11.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.865+0000 7f8ec603e700 1 Processor -- start 2023-03-03T06:18:11.867 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.865+0000 7f6f5ae1f700 1 -- start start 2023-03-03T06:18:11.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.865+0000 7f8ec603e700 1 -- start start 2023-03-03T06:18:11.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.865+0000 7f814b61e700 1 Processor -- start 2023-03-03T06:18:11.868 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f8ec603e700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ec00b5fd0 0x7f8ec014f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f8ec603e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec00bc4a0 con 0x7f8ec0145510 2023-03-03T06:18:11.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f8ec603e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec01458e0 con 0x7f8ec00bd3f0 2023-03-03T06:18:11.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f8ec603e700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec00b7e40 con 0x7f8ec00b5fd0 2023-03-03T06:18:11.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f6f5ae1f700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6f540c1760 0x7f6f5414f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.869 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f6f5ae1f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540b5fd0 0x7f6f540bc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f6f5ae1f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f540bc4a0 con 0x7f6f540bd3f0 2023-03-03T06:18:11.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f6f5ae1f700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f540b7e40 con 0x7f6f540c1760 2023-03-03T06:18:11.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f6f5ae1f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5414f6d0 con 0x7f6f540b5fd0 2023-03-03T06:18:11.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8ebeffd700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ec00b5fd0 0x7f8ec014f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.870 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8ebeffd700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ec00b5fd0 0x7f8ec014f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34212/0 (socket says 172.21.15.90:34212) 2023-03-03T06:18:11.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8ebeffd700 1 -- 172.21.15.90:0/2369179574 learned_addr learned my addr 172.21.15.90:0/2369179574 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.866+0000 7f814b61e700 1 -- start start 2023-03-03T06:18:11.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f6f593bc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6f540bd3f0 0x7f6f541452a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:46058/0 (socket says 172.21.15.90:46058) 2023-03-03T06:18:11.871 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f6f593bc700 1 -- 172.21.15.90:0/768015693 learned_addr learned my addr 172.21.15.90:0/768015693 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:11.872 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f814b61e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f81440bb9d0 0x7f8144153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.872 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f814b61e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81440b81f0 con 0x7f81440a6e80 2023-03-03T06:18:11.872 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f814b61e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81440b7f10 con 0x7f81440bad50 2023-03-03T06:18:11.872 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f814b61e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81440b6310 con 0x7f81440bb9d0 2023-03-03T06:18:11.872 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8149bbb700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f81440a6e80 0x7f814415dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53430/0 (socket says 172.21.15.90:53430) 2023-03-03T06:18:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8149bbb700 1 -- 172.21.15.90:0/3607279191 learned_addr learned my addr 172.21.15.90:0/3607279191 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8148bb9700 1 --2- 172.21.15.90:0/3607279191 >> v2:172.21.15.121:3300/0 conn(0x7f81440bb9d0 0x7f8144153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f6f58bbb700 1 --2- 172.21.15.90:0/768015693 >> v2:172.21.15.90:3300/0 conn(0x7f6f540c1760 0x7f6f5414f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8ebeffd700 1 -- 172.21.15.90:0/2369179574 >> v1:172.21.15.121:6789/0 conn(0x7f8ec00bd3f0 legacy=0x7f8ec0155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8ebeffd700 1 -- 172.21.15.90:0/2369179574 >> v1:172.21.15.90:6789/0 conn(0x7f8ec0145510 legacy=0x7f8ec006c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f8ebeffd700 1 -- 172.21.15.90:0/2369179574 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00bbb70 con 0x7f8ec00b5fd0 2023-03-03T06:18:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f58bbb700 1 -- 172.21.15.90:0/768015693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540b5fd0 msgr2=0x7f6f540bc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f58bbb700 1 --2- 172.21.15.90:0/768015693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540b5fd0 0x7f6f540bc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f8ebeffd700 1 --2- 172.21.15.90:0/2369179574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00b5fd0 0x7f8ec014f0f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4007360 tx=0x7f8eb4007670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76ddabb5cb2481b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f58bbb700 1 -- 172.21.15.90:0/768015693 >> v1:172.21.15.121:6789/0 conn(0x7f6f540bd3f0 legacy=0x7f6f541452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.875 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f58bbb700 1 -- 172.21.15.90:0/768015693 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f540bbe50 con 0x7f6f540c1760 2023-03-03T06:18:11.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f8ebe7fc700 1 -- 172.21.15.90:0/2369179574 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400a580 con 0x7f8ec00b5fd0 2023-03-03T06:18:11.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f8ebe7fc700 1 -- 172.21.15.90:0/2369179574 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400a6e0 con 0x7f8ec00b5fd0 2023-03-03T06:18:11.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f8ebe7fc700 1 -- 172.21.15.90:0/2369179574 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4004230 con 0x7f8ec00b5fd0 2023-03-03T06:18:11.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f58bbb700 1 --2- 172.21.15.90:0/768015693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540c1760 0x7f6f5414f0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009870 tx=0x7f6f48009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34701f89088cf1b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.876 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f8148bb9700 1 -- 172.21.15.90:0/3607279191 >> v1:172.21.15.121:6789/0 conn(0x7f81440bad50 legacy=0x7f81440c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f8148bb9700 1 -- 172.21.15.90:0/3607279191 >> v1:172.21.15.90:6789/0 conn(0x7f81440a6e80 legacy=0x7f814415dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f8148bb9700 1 -- 172.21.15.90:0/3607279191 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81440b65f0 con 0x7f81440bb9d0 2023-03-03T06:18:11.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f537fe700 1 -- 172.21.15.90:0/768015693 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48011830 con 0x7f6f540c1760 2023-03-03T06:18:11.877 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f537fe700 1 -- 172.21.15.90:0/768015693 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48011990 con 0x7f6f540c1760 2023-03-03T06:18:11.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f6f537fe700 1 -- 172.21.15.90:0/768015693 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48011d00 con 0x7f6f540c1760 2023-03-03T06:18:11.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f8ec603e700 1 -- 172.21.15.90:0/2369179574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00b5fd0 msgr2=0x7f8ec014f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f8ec603e700 1 --2- 172.21.15.90:0/2369179574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00b5fd0 0x7f8ec014f0f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4007360 tx=0x7f8eb4007670 comp rx=0 tx=0).stop 2023-03-03T06:18:11.878 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f8148bb9700 1 --2- 172.21.15.90:0/3607279191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440bb9d0 0x7f8144153560 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8140009840 tx=0x7f8140009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e9b30b3a2ca4c35 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.879 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f97d6796700 1 Processor -- start 2023-03-03T06:18:11.879 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f106ed08700 1 Processor -- start 2023-03-03T06:18:11.879 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.867+0000 7f106ed08700 1 -- start start 2023-03-03T06:18:11.879 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f106ed08700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f10680bc1f0 0x7f106815f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.880 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f106ed08700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106815fbe0 0x7f1068160040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.880 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/768015693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540c1760 msgr2=0x7f6f5414f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.880 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/768015693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540c1760 0x7f6f5414f0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009870 tx=0x7f6f48009b80 comp rx=0 tx=0).stop 2023-03-03T06:18:11.880 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f106ed08700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10680b9310 con 0x7f10680c19d0 2023-03-03T06:18:11.881 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f106ed08700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10680bc9d0 con 0x7f10680bc1f0 2023-03-03T06:18:11.881 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.868+0000 7f106ed08700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10680bc6f0 con 0x7f106815fbe0 2023-03-03T06:18:11.881 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f106d2a5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106815fbe0 0x7f1068160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.881 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f1067fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f10680bc1f0 0x7f106815f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.881 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f106d2a5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106815fbe0 0x7f1068160040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39162/0 (socket says 172.21.15.90:39162) 2023-03-03T06:18:11.881 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f1067fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f10680bc1f0 0x7f106815f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48370/0 (socket says 172.21.15.90:48370) 2023-03-03T06:18:11.882 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f8ec603e700 1 -- 172.21.15.90:0/2369179574 shutdown_connections 2023-03-03T06:18:11.882 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f106d2a5700 1 -- 172.21.15.90:0/3108254311 learned_addr learned my addr 172.21.15.90:0/3108254311 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.882 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f8ec603e700 1 --2- 172.21.15.90:0/2369179574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00b5fd0 0x7f8ec014f0f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.882 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f8ec603e700 1 -- 172.21.15.90:0/2369179574 >> 172.21.15.90:0/2369179574 conn(0x7f8ec006afc0 msgr2=0x7f8ec00c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f106d2a5700 1 -- 172.21.15.90:0/3108254311 >> v2:172.21.15.121:3300/0 conn(0x7f10680bc1f0 msgr2=0x7f106815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f106d2a5700 1 --2- 172.21.15.90:0/3108254311 >> v2:172.21.15.121:3300/0 conn(0x7f10680bc1f0 0x7f106815f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f106d2a5700 1 -- 172.21.15.90:0/3108254311 >> v1:172.21.15.121:6789/0 conn(0x7f10680c19d0 legacy=0x7f10680b8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f106d2a5700 1 -- 172.21.15.90:0/3108254311 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10680b6920 con 0x7f106815fbe0 2023-03-03T06:18:11.883 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f97d6796700 1 -- start start 2023-03-03T06:18:11.884 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f813bfff700 1 -- 172.21.15.90:0/3607279191 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140004230 con 0x7f81440bb9d0 2023-03-03T06:18:11.884 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f813bfff700 1 -- 172.21.15.90:0/3607279191 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8140004390 con 0x7f81440bb9d0 2023-03-03T06:18:11.884 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f813bfff700 1 -- 172.21.15.90:0/3607279191 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81400036a0 con 0x7f81440bb9d0 2023-03-03T06:18:11.884 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.869+0000 7f10677fe700 1 -- 172.21.15.90:0/3108254311 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282661759 0 0) 0x7f10680b9310 con 0x7f10680c19d0 2023-03-03T06:18:11.885 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f106d2a5700 1 --2- 172.21.15.90:0/3108254311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106815fbe0 0x7f1068160040 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1060008ec0 tx=0x7f106000e640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=411cc3978a6859ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.885 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f8ec603e700 1 -- 172.21.15.90:0/2369179574 shutdown_connections 2023-03-03T06:18:11.885 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f8ec603e700 1 -- 172.21.15.90:0/2369179574 wait complete. 2023-03-03T06:18:11.886 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f10677fe700 1 -- 172.21.15.90:0/3108254311 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060010640 con 0x7f106815fbe0 2023-03-03T06:18:11.886 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f10677fe700 1 -- 172.21.15.90:0/3108254311 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10600107a0 con 0x7f106815fbe0 2023-03-03T06:18:11.886 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f10677fe700 1 -- 172.21.15.90:0/3108254311 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060010b30 con 0x7f106815fbe0 2023-03-03T06:18:11.886 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f97d6796700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f97d00c2910 0x7f97d00c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.887 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/768015693 shutdown_connections 2023-03-03T06:18:11.887 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/768015693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540c1760 0x7f6f5414f0f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.887 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/768015693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540b5fd0 0x7f6f540bc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.887 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/768015693 >> 172.21.15.90:0/768015693 conn(0x7f6f5406afc0 msgr2=0x7f6f540c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.887 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f97d6796700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d00bfe90 0x7f97d0149310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.888 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f97d6796700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00c0640 con 0x7f97d0149850 2023-03-03T06:18:11.888 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f97d6796700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d006b3b0 con 0x7f97d00c2910 2023-03-03T06:18:11.888 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.870+0000 7f97d6796700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00c1d60 con 0x7f97d00bfe90 2023-03-03T06:18:11.888 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f1b82e4d700 1 Processor -- start 2023-03-03T06:18:11.888 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f97cf7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d00bfe90 0x7f97d0149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.889 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f97cf7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d00bfe90 0x7f97d0149310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39170/0 (socket says 172.21.15.90:39170) 2023-03-03T06:18:11.889 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f97cf7fe700 1 -- 172.21.15.90:0/3181618146 learned_addr learned my addr 172.21.15.90:0/3181618146 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.889 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f1b82e4d700 1 -- start start 2023-03-03T06:18:11.889 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/768015693 shutdown_connections 2023-03-03T06:18:11.890 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f106ed08700 1 -- 172.21.15.90:0/3108254311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106815fbe0 msgr2=0x7f1068160040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.890 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f106ed08700 1 --2- 172.21.15.90:0/3108254311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106815fbe0 0x7f1068160040 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1060008ec0 tx=0x7f106000e640 comp rx=0 tx=0).stop 2023-03-03T06:18:11.890 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f106ed08700 1 -- 172.21.15.90:0/3108254311 shutdown_connections 2023-03-03T06:18:11.890 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f106ed08700 1 --2- 172.21.15.90:0/3108254311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106815fbe0 0x7f1068160040 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.890 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f106ed08700 1 --2- 172.21.15.90:0/3108254311 >> v2:172.21.15.121:3300/0 conn(0x7f10680bc1f0 0x7f106815f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.891 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f106ed08700 1 -- 172.21.15.90:0/3108254311 >> 172.21.15.90:0/3108254311 conn(0x7f106806afc0 msgr2=0x7f10680c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.891 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f97cf7fe700 1 -- 172.21.15.90:0/3181618146 >> v2:172.21.15.90:3300/0 conn(0x7f97d00c2910 msgr2=0x7f97d00c0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.891 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f97cf7fe700 1 --2- 172.21.15.90:0/3181618146 >> v2:172.21.15.90:3300/0 conn(0x7f97d00c2910 0x7f97d00c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.891 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f97cf7fe700 1 -- 172.21.15.90:0/3181618146 >> v1:172.21.15.90:6789/0 conn(0x7f97d0149850 legacy=0x7f97d006a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.891 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f97cf7fe700 1 -- 172.21.15.90:0/3181618146 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d00c0920 con 0x7f97d00bfe90 2023-03-03T06:18:11.892 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f8ec603e700 1 Processor -- start 2023-03-03T06:18:11.892 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.871+0000 7f106ed08700 1 -- 172.21.15.90:0/3108254311 shutdown_connections 2023-03-03T06:18:11.892 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f97cf7fe700 1 --2- 172.21.15.90:0/3181618146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d00bfe90 0x7f97d0149310 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009e50 tx=0x7f97c800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f20fbbb805c35f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.893 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f106ed08700 1 -- 172.21.15.90:0/3108254311 wait complete. 2023-03-03T06:18:11.893 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f97ceffd700 1 -- 172.21.15.90:0/3181618146 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c800e070 con 0x7f97d00bfe90 2023-03-03T06:18:11.894 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f8ec603e700 1 -- start start 2023-03-03T06:18:11.894 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f97ceffd700 1 -- 172.21.15.90:0/3181618146 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c8003a20 con 0x7f97d00bfe90 2023-03-03T06:18:11.894 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f97ceffd700 1 -- 172.21.15.90:0/3181618146 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8003dd0 con 0x7f97d00bfe90 2023-03-03T06:18:11.894 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/768015693 wait complete. 2023-03-03T06:18:11.895 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b82e4d700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1b7c0b7b80 0x7f1b7c06c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.895 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b82e4d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c06cde0 0x7f1b7c14bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.895 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b82e4d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c14f550 con 0x7f1b7c0bb280 2023-03-03T06:18:11.895 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b82e4d700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7c0c1e70 con 0x7f1b7c0b7b80 2023-03-03T06:18:11.895 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b82e4d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0bcb00 con 0x7f1b7c06cde0 2023-03-03T06:18:11.895 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b813ea700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c06cde0 0x7f1b7c14bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.896 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b813ea700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c06cde0 0x7f1b7c14bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39174/0 (socket says 172.21.15.90:39174) 2023-03-03T06:18:11.896 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f1b813ea700 1 -- 172.21.15.90:0/506307172 learned_addr learned my addr 172.21.15.90:0/506307172 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.896 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f97d6796700 1 -- 172.21.15.90:0/3181618146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d00bfe90 msgr2=0x7f97d0149310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.896 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f97d6796700 1 --2- 172.21.15.90:0/3181618146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d00bfe90 0x7f97d0149310 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009e50 tx=0x7f97c800b330 comp rx=0 tx=0).stop 2023-03-03T06:18:11.897 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b813ea700 1 -- 172.21.15.90:0/506307172 >> v2:172.21.15.90:3300/0 conn(0x7f1b7c0b7b80 msgr2=0x7f1b7c06c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.897 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b813ea700 1 --2- 172.21.15.90:0/506307172 >> v2:172.21.15.90:3300/0 conn(0x7f1b7c0b7b80 0x7f1b7c06c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.897 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b813ea700 1 -- 172.21.15.90:0/506307172 >> v1:172.21.15.90:6789/0 conn(0x7f1b7c0bb280 legacy=0x7f1b7c0c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.897 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b813ea700 1 -- 172.21.15.90:0/506307172 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0a6f20 con 0x7f1b7c06cde0 2023-03-03T06:18:11.897 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ec603e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ec00b5fd0 0x7f8ec01e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ec603e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ec00bd3f0 0x7f8ec01e53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ec603e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec0145510 0x7f8ec0071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ec603e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec0071e40 con 0x7f8ec0145510 2023-03-03T06:18:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ec603e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec014b3f0 con 0x7f8ec00bd3f0 2023-03-03T06:18:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ec603e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec014b6d0 con 0x7f8ec00b5fd0 2023-03-03T06:18:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec0145510 0x7f8ec0071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.899 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec0145510 0x7f8ec0071900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34244/0 (socket says 172.21.15.90:34244) 2023-03-03T06:18:11.899 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 -- 172.21.15.90:0/82684619 learned_addr learned my addr 172.21.15.90:0/82684619 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.899 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b813ea700 1 --2- 172.21.15.90:0/506307172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c06cde0 0x7f1b7c14bbd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009e50 tx=0x7f1b7400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4128c1a2b5144 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.899 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f814b61e700 1 -- 172.21.15.90:0/3607279191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440bb9d0 msgr2=0x7f8144153560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.899 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.872+0000 7f814b61e700 1 --2- 172.21.15.90:0/3607279191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440bb9d0 0x7f8144153560 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8140009840 tx=0x7f8140009b50 comp rx=0 tx=0).stop 2023-03-03T06:18:11.900 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f106ed08700 1 Processor -- start 2023-03-03T06:18:11.900 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f814b61e700 1 -- 172.21.15.90:0/3607279191 shutdown_connections 2023-03-03T06:18:11.900 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f814b61e700 1 --2- 172.21.15.90:0/3607279191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440bb9d0 0x7f8144153560 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.900 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f814b61e700 1 -- 172.21.15.90:0/3607279191 >> 172.21.15.90:0/3607279191 conn(0x7f814406afc0 msgr2=0x7f81440c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.901 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b7b7fe700 1 -- 172.21.15.90:0/506307172 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400e070 con 0x7f1b7c06cde0 2023-03-03T06:18:11.901 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b7b7fe700 1 -- 172.21.15.90:0/506307172 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b74003a20 con 0x7f1b7c06cde0 2023-03-03T06:18:11.901 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f1b7b7fe700 1 -- 172.21.15.90:0/506307172 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74003dd0 con 0x7f1b7c06cde0 2023-03-03T06:18:11.901 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 -- 172.21.15.90:0/82684619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ec00b5fd0 msgr2=0x7f8ec01e7e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.902 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ec00b5fd0 0x7f8ec01e7e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.902 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 -- 172.21.15.90:0/82684619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ec00bd3f0 msgr2=0x7f8ec01e53e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.902 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ec00bd3f0 0x7f8ec01e53e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.902 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f8ebffff700 1 -- 172.21.15.90:0/82684619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4007040 con 0x7f8ec0145510 2023-03-03T06:18:11.902 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f6f5ae1f700 1 Processor -- start 2023-03-03T06:18:11.903 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f8ebffff700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec0145510 0x7f8ec0071900 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000b970 tx=0x7f8eb000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.903 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f814b61e700 1 -- 172.21.15.90:0/3607279191 shutdown_connections 2023-03-03T06:18:11.903 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f6f5ae1f700 1 -- start start 2023-03-03T06:18:11.903 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f5ae1f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540b5fd0 0x7f6f540c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f5ae1f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540bd3f0 0x7f6f541479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f5ae1f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f540c1760 0x7f6f54147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f5ae1f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5414a8a0 con 0x7f6f540b5fd0 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f5ae1f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f54139560 con 0x7f6f540c1760 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f5ae1f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540a6ba0 con 0x7f6f540bd3f0 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f540c1760 0x7f6f54147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f540c1760 0x7f6f54147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48392/0 (socket says 172.21.15.90:48392) 2023-03-03T06:18:11.904 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 -- 172.21.15.90:0/3921362893 learned_addr learned my addr 172.21.15.90:0/3921362893 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.905 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f53fff700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540bd3f0 0x7f6f541479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.905 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f8eaffff700 1 -- 172.21.15.90:0/82684619 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000c9b0 con 0x7f8ec0145510 2023-03-03T06:18:11.905 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f8eaffff700 1 -- 172.21.15.90:0/82684619 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb000cb10 con 0x7f8ec0145510 2023-03-03T06:18:11.905 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f8eaffff700 1 -- 172.21.15.90:0/82684619 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0014440 con 0x7f8ec0145510 2023-03-03T06:18:11.905 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec00bc130 con 0x7f8ec0145510 2023-03-03T06:18:11.905 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec00bbe50 con 0x7f8ec0145510 2023-03-03T06:18:11.906 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f814b61e700 1 -- 172.21.15.90:0/3607279191 wait complete. 2023-03-03T06:18:11.906 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f814b61e700 1 Processor -- start 2023-03-03T06:18:11.906 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f1b82e4d700 1 -- 172.21.15.90:0/506307172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c06cde0 msgr2=0x7f1b7c14bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.906 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/506307172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c06cde0 0x7f1b7c14bbd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009e50 tx=0x7f1b7400b330 comp rx=0 tx=0).stop 2023-03-03T06:18:11.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f1b82e4d700 1 -- 172.21.15.90:0/506307172 shutdown_connections 2023-03-03T06:18:11.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/506307172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c06cde0 0x7f1b7c14bbd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/506307172 >> v2:172.21.15.90:3300/0 conn(0x7f1b7c0b7b80 0x7f1b7c06c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.874+0000 7f1b82e4d700 1 -- 172.21.15.90:0/506307172 >> 172.21.15.90:0/506307172 conn(0x7f1b7c06afc0 msgr2=0x7f1b7c0c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1b82e4d700 1 -- 172.21.15.90:0/506307172 shutdown_connections 2023-03-03T06:18:11.907 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f97d6796700 1 -- 172.21.15.90:0/3181618146 shutdown_connections 2023-03-03T06:18:11.908 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f97d6796700 1 --2- 172.21.15.90:0/3181618146 >> v2:172.21.15.90:3300/0 conn(0x7f97d00c2910 0x7f97d00c0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.908 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f97d6796700 1 --2- 172.21.15.90:0/3181618146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d00bfe90 0x7f97d0149310 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.908 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f97d6796700 1 -- 172.21.15.90:0/3181618146 >> 172.21.15.90:0/3181618146 conn(0x7f97d00c61a0 msgr2=0x7f97d00c1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.908 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.873+0000 7f106ed08700 1 -- start start 2023-03-03T06:18:11.908 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f106ed08700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10680bc1f0 0x7f1068070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.908 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f106ed08700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10680c19d0 0x7f1068071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f106ed08700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f106815fbe0 0x7f10680bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f106ed08700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680bf870 con 0x7f106815fbe0 2023-03-03T06:18:11.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f106ed08700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680bf620 con 0x7f10680bc1f0 2023-03-03T06:18:11.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f106ed08700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106806f990 con 0x7f10680c19d0 2023-03-03T06:18:11.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10680c19d0 0x7f1068071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10680c19d0 0x7f1068071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39190/0 (socket says 172.21.15.90:39190) 2023-03-03T06:18:11.909 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 -- 172.21.15.90:0/1799631576 learned_addr learned my addr 172.21.15.90:0/1799631576 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 -- 172.21.15.90:0/1799631576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10680bc1f0 msgr2=0x7f1068070100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-03T06:18:11.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10680bc1f0 0x7f1068070100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 -- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f106815fbe0 msgr2=0x7f10680bfa10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-03T06:18:11.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f106815fbe0 0x7f10680bfa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1067fff700 1 -- 172.21.15.90:0/1799631576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060008b20 con 0x7f10680c19d0 2023-03-03T06:18:11.910 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f1067fff700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10680c19d0 0x7f1068071050 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f105800e9b0 tx=0x7f105800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f97d6796700 1 -- 172.21.15.90:0/3181618146 shutdown_connections 2023-03-03T06:18:11.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f1b82e4d700 1 -- 172.21.15.90:0/506307172 wait complete. 2023-03-03T06:18:11.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 -- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540bd3f0 msgr2=0x7f6f541479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.911 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540bd3f0 0x7f6f541479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 -- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540b5fd0 msgr2=0x7f6f540c07c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-03T06:18:11.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540b5fd0 0x7f6f540c07c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.875+0000 7f6f593bc700 1 -- 172.21.15.90:0/3921362893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f480094a0 con 0x7f6f540c1760 2023-03-03T06:18:11.912 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f1b82e4d700 1 Processor -- start 2023-03-03T06:18:11.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f10657fa700 1 -- 172.21.15.90:0/1799631576 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800c850 con 0x7f10680c19d0 2023-03-03T06:18:11.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f10657fa700 1 -- 172.21.15.90:0/1799631576 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800c9b0 con 0x7f10680c19d0 2023-03-03T06:18:11.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f10657fa700 1 -- 172.21.15.90:0/1799631576 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800cd20 con 0x7f10680c19d0 2023-03-03T06:18:11.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10680b6ee0 con 0x7f10680c19d0 2023-03-03T06:18:11.913 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10680b6c00 con 0x7f10680c19d0 2023-03-03T06:18:11.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f6f593bc700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f540c1760 0x7f6f54147f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400c450 tx=0x7f6f4400c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f814b61e700 1 -- start start 2023-03-03T06:18:11.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f814b61e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440a6e80 0x7f81440c00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f814b61e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81440bad50 0x7f814414f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f814b61e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81440bb9d0 0x7f814414dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.914 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f814b61e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814414a8b0 con 0x7f81440bad50 2023-03-03T06:18:11.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f814b61e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8144150c00 con 0x7f81440a6e80 2023-03-03T06:18:11.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f814b61e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81440c0630 con 0x7f81440bb9d0 2023-03-03T06:18:11.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f97d6796700 1 -- 172.21.15.90:0/3181618146 wait complete. 2023-03-03T06:18:11.915 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f81493ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440a6e80 0x7f81440c00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f81493ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440a6e80 0x7f81440c00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48398/0 (socket says 172.21.15.90:48398) 2023-03-03T06:18:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f81493ba700 1 -- 172.21.15.90:0/2734409511 learned_addr learned my addr 172.21.15.90:0/2734409511 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f8149bbb700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81440bb9d0 0x7f814414dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f97d6796700 1 Processor -- start 2023-03-03T06:18:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.876+0000 7f1b82e4d700 1 -- start start 2023-03-03T06:18:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f1b82e4d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b7c06cde0 0x7f1b7c1f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f1b82e4d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b7c0b7b80 0x7f1b7c1f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f1b82e4d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c0bb280 0x7f1b7c1f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f1b82e4d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0be1e0 con 0x7f1b7c06cde0 2023-03-03T06:18:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f1b82e4d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c1e43b0 con 0x7f1b7c0b7b80 2023-03-03T06:18:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f1b82e4d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c1f54c0 con 0x7f1b7c0bb280 2023-03-03T06:18:11.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f6f517fa700 1 -- 172.21.15.90:0/3921362893 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44005d70 con 0x7f6f540c1760 2023-03-03T06:18:11.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f6f517fa700 1 -- 172.21.15.90:0/3921362893 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4400ce70 con 0x7f6f540c1760 2023-03-03T06:18:11.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f6f517fa700 1 -- 172.21.15.90:0/3921362893 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f440072d0 con 0x7f6f540c1760 2023-03-03T06:18:11.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5406d7f0 con 0x7f6f540c1760 2023-03-03T06:18:11.918 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f540bc130 con 0x7f6f540c1760 2023-03-03T06:18:11.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f97d6796700 1 -- start start 2023-03-03T06:18:11.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.877+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec006d7f0 con 0x7f8ec0145510 2023-03-03T06:18:11.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8eaffff700 1 -- 172.21.15.90:0/82684619 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8eb001b030 con 0x7f8ec0145510 2023-03-03T06:18:11.919 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8149bbb700 1 -- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440a6e80 msgr2=0x7f81440c00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8149bbb700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440a6e80 0x7f81440c00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8149bbb700 1 -- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81440bad50 msgr2=0x7f814414f6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-03T06:18:11.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8149bbb700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81440bad50 0x7f814414f6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8149bbb700 1 -- 172.21.15.90:0/2734409511 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81400094a0 con 0x7f81440bb9d0 2023-03-03T06:18:11.920 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8149bbb700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81440bb9d0 0x7f814414dea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f813c00cba0 tx=0x7f813c00ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b7c06cde0 0x7f1b7c1f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b7c06cde0 0x7f1b7c1f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34260/0 (socket says 172.21.15.90:34260) 2023-03-03T06:18:11.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 -- 172.21.15.90:0/157004214 learned_addr learned my addr 172.21.15.90:0/157004214 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 -- 172.21.15.90:0/157004214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c0bb280 msgr2=0x7f1b7c1f2e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c0bb280 0x7f1b7c1f2e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 -- 172.21.15.90:0/157004214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b7c0b7b80 msgr2=0x7f1b7c1f28f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.921 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b7c0b7b80 0x7f1b7c1f28f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 -- 172.21.15.90:0/157004214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b74009b30 con 0x7f1b7c06cde0 2023-03-03T06:18:11.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f1b80be9700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b7c06cde0 0x7f1b7c1f4bf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000c4d0 tx=0x7f1b7000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f8eaffff700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f8ea4068390 0x7f8ea406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.922 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f8eaffff700 1 -- 172.21.15.90:0/82684619 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8eb00859e0 con 0x7f8ec0145510 2023-03-03T06:18:11.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f97d6796700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d00bfe90 0x7f97d00b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f97d6796700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d00c2910 0x7f97d01ecb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f97d6796700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d0149850 0x7f97d00a8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f97d6796700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d01e3300 con 0x7f97d00bfe90 2023-03-03T06:18:11.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f97d6796700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00a83e0 con 0x7f97d00c2910 2023-03-03T06:18:11.923 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f97d6796700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00be9b0 con 0x7f97d0149850 2023-03-03T06:18:11.924 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f8ebf7fe700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f8ea4068390 0x7f8ea406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.924 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8139ffb700 1 -- 172.21.15.90:0/2734409511 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c00aad0 con 0x7f81440bb9d0 2023-03-03T06:18:11.924 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8139ffb700 1 -- 172.21.15.90:0/2734409511 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c00ac30 con 0x7f81440bb9d0 2023-03-03T06:18:11.924 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f8139ffb700 1 -- 172.21.15.90:0/2734409511 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c0070d0 con 0x7f81440bb9d0 2023-03-03T06:18:11.924 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81440c1e70 con 0x7f81440bb9d0 2023-03-03T06:18:11.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.878+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81440b68d0 con 0x7f81440bb9d0 2023-03-03T06:18:11.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81440bb120 con 0x7f81440bb9d0 2023-03-03T06:18:11.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d00bfe90 0x7f97d00b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d00bfe90 0x7f97d00b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34268/0 (socket says 172.21.15.90:34268) 2023-03-03T06:18:11.925 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 -- 172.21.15.90:0/3766344974 learned_addr learned my addr 172.21.15.90:0/3766344974 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f8ebf7fe700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f8ea4068390 0x7f8ea406a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000f60 tx=0x7f8eb8007a40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 -- 172.21.15.90:0/3766344974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d0149850 msgr2=0x7f97d00a8830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d0149850 0x7f97d00a8830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.926 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 -- 172.21.15.90:0/3766344974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d00c2910 msgr2=0x7f97d01ecb50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.927 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d00c2910 0x7f97d01ecb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.927 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f97cffff700 1 -- 172.21.15.90:0/3766344974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8009b30 con 0x7f97d00bfe90 2023-03-03T06:18:11.927 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f97cffff700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d00bfe90 0x7f97d00b4360 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f97c000b530 tx=0x7f97c000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.927 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f6f517fa700 1 -- 172.21.15.90:0/3921362893 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f44019030 con 0x7f6f540c1760 2023-03-03T06:18:11.927 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.879+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10480052e0 con 0x7f10680c19d0 2023-03-03T06:18:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.880+0000 7f10657fa700 1 -- 172.21.15.90:0/1799631576 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1058023070 con 0x7f10680c19d0 2023-03-03T06:18:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f10657fa700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f10540682f0 0x7f105406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f10657fa700 1 -- 172.21.15.90:0/1799631576 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1058086020 con 0x7f10680c19d0 2023-03-03T06:18:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f106caa4700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f10540682f0 0x7f105406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f6f517fa700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6f40068340 0x7f6f4006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.929 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f6f58bbb700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6f40068340 0x7f6f4006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.929 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f6f517fa700 1 -- 172.21.15.90:0/3921362893 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6f44084ee0 con 0x7f6f540c1760 2023-03-03T06:18:11.929 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f106caa4700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f10540682f0 0x7f105406a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f105c007fd0 tx=0x7f105c007940 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.929 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.881+0000 7f8139ffb700 1 -- 172.21.15.90:0/2734409511 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f813c01a030 con 0x7f81440bb9d0 2023-03-03T06:18:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.882+0000 7f6f58bbb700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6f40068340 0x7f6f4006a7f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6f540b86d0 tx=0x7f6f48008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.882+0000 7f8139ffb700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81280683c0 0x7f812806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.882+0000 7f81493ba700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81280683c0 0x7f812806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.882+0000 7f8139ffb700 1 -- 172.21.15.90:0/2734409511 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f813c0861e0 con 0x7f81440bb9d0 2023-03-03T06:18:11.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.882+0000 7f81493ba700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81280683c0 0x7f812806a870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f81340013c0 tx=0x7f8134009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.883+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5406c8a0 con 0x7f6f540c1760 2023-03-03T06:18:11.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.885+0000 7fe0bd7da700 1 Processor -- start 2023-03-03T06:18:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.885+0000 7f8139ffb700 1 -- 172.21.15.90:0/2734409511 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f813c052a80 con 0x7f81440bb9d0 2023-03-03T06:18:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.885+0000 7fe0bd7da700 1 -- start start 2023-03-03T06:18:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.885+0000 7f634fdce700 1 Processor -- start 2023-03-03T06:18:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.885+0000 7f634fdce700 1 -- start start 2023-03-03T06:18:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0bd7da700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b8149530 0x7fe0b806be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0bd7da700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe0b806ee30 0x7fe0b806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0bd7da700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b8152910 con 0x7fe0b806e9c0 2023-03-03T06:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0bd7da700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b814fd60 con 0x7fe0b806ee30 2023-03-03T06:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0bd7da700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b806c340 con 0x7fe0b8149530 2023-03-03T06:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0b67fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe0b806e9c0 0x7fe0b806c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53488/0 (socket says 172.21.15.90:53488) 2023-03-03T06:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0b67fc700 1 -- 172.21.15.90:0/1009715261 learned_addr learned my addr 172.21.15.90:0/1009715261 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0b6ffd700 1 --2- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b8149530 0x7fe0b806be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7fe0b77fe700 1 --2- 172.21.15.90:0/1009715261 >> v2:172.21.15.121:3300/0 conn(0x7fe0b806ee30 0x7fe0b806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f1b797fa700 1 -- 172.21.15.90:0/157004214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b700054e0 con 0x7f1b7c06cde0 2023-03-03T06:18:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634fdce700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f634806a330 0x7f63480bd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634fdce700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f634806a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634fdce700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63480bcae0 con 0x7f6348152770 2023-03-03T06:18:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634fdce700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634806d920 con 0x7f634806a330 2023-03-03T06:18:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634fdce700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63480bcd30 con 0x7f634806e950 2023-03-03T06:18:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634db6a700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f634806a330 0x7f63480bd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634e36b700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6348152770 0x7f6348152b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:46138/0 (socket says 172.21.15.90:46138) 2023-03-03T06:18:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634e36b700 1 -- 172.21.15.90:0/2038717344 learned_addr learned my addr 172.21.15.90:0/2038717344 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f634db6a700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f634806a330 0x7f63480bd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34286/0 (socket says 172.21.15.90:34286) 2023-03-03T06:18:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f8eaffff700 1 -- 172.21.15.90:0/82684619 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8eb00521d0 con 0x7f8ec0145510 2023-03-03T06:18:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f97ccff9700 1 -- 172.21.15.90:0/3766344974 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c0011830 con 0x7f97d00bfe90 2023-03-03T06:18:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f634d369700 1 --2- 172.21.15.90:0/2038717344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f634806a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f97ccff9700 1 -- 172.21.15.90:0/3766344974 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c0011990 con 0x7f97d00bfe90 2023-03-03T06:18:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d00c1980 con 0x7f97d00bfe90 2023-03-03T06:18:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f634d369700 1 -- 172.21.15.90:0/2038717344 >> v2:172.21.15.90:3300/0 conn(0x7f634806a330 msgr2=0x7f63480bd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f634d369700 1 --2- 172.21.15.90:0/2038717344 >> v2:172.21.15.90:3300/0 conn(0x7f634806a330 0x7f63480bd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f634d369700 1 -- 172.21.15.90:0/2038717344 >> v1:172.21.15.121:6789/0 conn(0x7f6348152770 legacy=0x7f6348152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f634d369700 1 -- 172.21.15.90:0/2038717344 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634814fd20 con 0x7f634806e950 2023-03-03T06:18:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d00c16a0 con 0x7f97d00bfe90 2023-03-03T06:18:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f97ccff9700 1 -- 172.21.15.90:0/3766344974 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c0011d00 con 0x7f97d00bfe90 2023-03-03T06:18:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f1b797fa700 1 -- 172.21.15.90:0/157004214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b70003680 con 0x7f1b7c06cde0 2023-03-03T06:18:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f1b797fa700 1 -- 172.21.15.90:0/157004214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70003a30 con 0x7f1b7c06cde0 2023-03-03T06:18:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c14b6b0 con 0x7f1b7c06cde0 2023-03-03T06:18:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.886+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c14f7b0 con 0x7f1b7c06cde0 2023-03-03T06:18:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7f634d369700 1 --2- 172.21.15.90:0/2038717344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f634806a700 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6338009840 tx=0x7f6338009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9f130442b6ed3d1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7fe0b77fe700 1 -- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b8149530 msgr2=0x7fe0b806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7fe0b77fe700 1 --2- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b8149530 0x7fe0b806be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7fe0b77fe700 1 -- 172.21.15.90:0/1009715261 >> v1:172.21.15.90:6789/0 conn(0x7fe0b806e9c0 legacy=0x7fe0b806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7fe0b77fe700 1 -- 172.21.15.90:0/1009715261 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b814f430 con 0x7fe0b806ee30 2023-03-03T06:18:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.887+0000 7fe0b77fe700 1 --2- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806ee30 0x7fe0b806d100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009d90 tx=0x7fe0b000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84750f1eb0954dba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.888+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7c14fa90 con 0x7f1b7c06cde0 2023-03-03T06:18:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.888+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b00052e0 con 0x7f97d00bfe90 2023-03-03T06:18:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.888+0000 7fe0b5ffb700 1 -- 172.21.15.90:0/1009715261 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b000e070 con 0x7fe0b806ee30 2023-03-03T06:18:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.888+0000 7fe0b5ffb700 1 -- 172.21.15.90:0/1009715261 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b000bb20 con 0x7fe0b806ee30 2023-03-03T06:18:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.888+0000 7fe0b5ffb700 1 -- 172.21.15.90:0/1009715261 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0012430 con 0x7fe0b806ee30 2023-03-03T06:18:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.889+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806ee30 msgr2=0x7fe0b806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.889+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806ee30 0x7fe0b806d100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009d90 tx=0x7fe0b000b330 comp rx=0 tx=0).stop 2023-03-03T06:18:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.889+0000 7f634cb68700 1 -- 172.21.15.90:0/2038717344 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338011830 con 0x7f634806e950 2023-03-03T06:18:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.889+0000 7f634cb68700 1 -- 172.21.15.90:0/2038717344 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6338011990 con 0x7f634806e950 2023-03-03T06:18:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.889+0000 7f634cb68700 1 -- 172.21.15.90:0/2038717344 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338011d20 con 0x7f634806e950 2023-03-03T06:18:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.889+0000 7f1b797fa700 1 -- 172.21.15.90:0/157004214 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b70010070 con 0x7f1b7c06cde0 2023-03-03T06:18:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f1b797fa700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1b600682f0 0x7f1b6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f1b797fa700 1 -- 172.21.15.90:0/157004214 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1b70084cd0 con 0x7f1b7c06cde0 2023-03-03T06:18:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f97ccff9700 1 -- 172.21.15.90:0/3766344974 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f97c0021070 con 0x7f97d00bfe90 2023-03-03T06:18:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f10657fa700 1 -- 172.21.15.90:0/1799631576 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1058052730 con 0x7f10680c19d0 2023-03-03T06:18:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f634fdce700 1 -- 172.21.15.90:0/2038717344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 msgr2=0x7f634806a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f634fdce700 1 --2- 172.21.15.90:0/2038717344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f634806a700 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6338009840 tx=0x7f6338009b50 comp rx=0 tx=0).stop 2023-03-03T06:18:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f97ccff9700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f97bc068340 0x7f97bc06a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1009715261 shutdown_connections 2023-03-03T06:18:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806ee30 0x7fe0b806d100 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1009715261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b8149530 0x7fe0b806be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1009715261 >> 172.21.15.90:0/1009715261 conn(0x7fe0b80c3270 msgr2=0x7fe0b80c3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f634fdce700 1 -- 172.21.15.90:0/2038717344 shutdown_connections 2023-03-03T06:18:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f634fdce700 1 --2- 172.21.15.90:0/2038717344 >> v2:172.21.15.90:3300/0 conn(0x7f634806a330 0x7f63480bd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f634fdce700 1 --2- 172.21.15.90:0/2038717344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f634806a700 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.890+0000 7f634fdce700 1 -- 172.21.15.90:0/2038717344 >> 172.21.15.90:0/2038717344 conn(0x7f634806b440 msgr2=0x7f634806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.891+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1009715261 shutdown_connections 2023-03-03T06:18:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.891+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1009715261 wait complete. 2023-03-03T06:18:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.891+0000 7f97ccff9700 1 -- 172.21.15.90:0/3766344974 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f97c0086110 con 0x7f97d00bfe90 2023-03-03T06:18:11.949 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.891+0000 7f634fdce700 1 -- 172.21.15.90:0/2038717344 shutdown_connections 2023-03-03T06:18:11.949 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.891+0000 7f634fdce700 1 -- 172.21.15.90:0/2038717344 wait complete. 2023-03-03T06:18:11.949 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.892+0000 7f634fdce700 1 Processor -- start 2023-03-03T06:18:11.950 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.892+0000 7fe0bd7da700 1 Processor -- start 2023-03-03T06:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.892+0000 7f634fdce700 1 -- start start 2023-03-03T06:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.893+0000 7f634fdce700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634806a330 0x7f6348150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634fdce700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f63480b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634fdce700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6348152770 0x7f6348148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634fdce700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6348150f90 con 0x7f6348152770 2023-03-03T06:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634fdce700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6348149be0 con 0x7f634806a330 2023-03-03T06:18:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634fdce700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63480c0500 con 0x7f634806e950 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f63480b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f63480b70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39234/0 (socket says 172.21.15.90:39234) 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 -- 172.21.15.90:0/2934999092 learned_addr learned my addr 172.21.15.90:0/2934999092 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 -- 172.21.15.90:0/2934999092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634806a330 msgr2=0x7f6348150760 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634806a330 0x7f6348150760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 -- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6348152770 msgr2=0x7f6348148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6348152770 0x7f6348148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 -- 172.21.15.90:0/2934999092 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63380094a0 con 0x7f634806e950 2023-03-03T06:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7f634d369700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f63480b70e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f63380090b0 tx=0x7f6338004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7f633e7fc700 1 -- 172.21.15.90:0/2934999092 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63380036a0 con 0x7f634806e950 2023-03-03T06:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7f633e7fc700 1 -- 172.21.15.90:0/2934999092 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633801f070 con 0x7f634806e950 2023-03-03T06:18:11.953 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7f633e7fc700 1 -- 172.21.15.90:0/2934999092 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338003a40 con 0x7f634806e950 2023-03-03T06:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.894+0000 7fe0bd7da700 1 -- start start 2023-03-03T06:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0bd7da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806e9c0 0x7fe0b80a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0bd7da700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b806ee30 0x7fe0b80a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0bd7da700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b8149530 0x7fe0b80a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0bd7da700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b806bf00 con 0x7fe0b8149530 2023-03-03T06:18:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0bd7da700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b814f210 con 0x7fe0b806e9c0 2023-03-03T06:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0bd7da700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b80c2b50 con 0x7fe0b806ee30 2023-03-03T06:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806e9c0 0x7fe0b80a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806e9c0 0x7fe0b80a8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.90:48444/0 (socket says 172.21.15.90:48444) 2023-03-03T06:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 -- 172.21.15.90:0/1540038023 learned_addr learned my addr 172.21.15.90:0/1540038023 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b67fc700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b806ee30 0x7fe0b80a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 -- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b806ee30 msgr2=0x7fe0b80a7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b806ee30 0x7fe0b80a7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 -- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b8149530 msgr2=0x7fe0b80a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b77fe700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b8149530 0x7fe0b80a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b8149530 0x7fe0b80a7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b6ffd700 1 -- 172.21.15.90:0/1540038023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b0009a70 con 0x7fe0b806e9c0 2023-03-03T06:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.895+0000 7fe0b77fe700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b8149530 0x7fe0b80a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63480bdde0 con 0x7f634806e950 2023-03-03T06:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7fe0b6ffd700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806e9c0 0x7fe0b80a8c00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800e9b0 tx=0x7fe0a800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7fe09f7fe700 1 -- 172.21.15.90:0/1540038023 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8009710 con 0x7fe0b806e9c0 2023-03-03T06:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7fe09f7fe700 1 -- 172.21.15.90:0/1540038023 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a800c850 con 0x7fe0b806e9c0 2023-03-03T06:18:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b814f9f0 con 0x7fe0b806e9c0 2023-03-03T06:18:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7fe09f7fe700 1 -- 172.21.15.90:0/1540038023 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a800cbc0 con 0x7fe0b806e9c0 2023-03-03T06:18:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b814f710 con 0x7fe0b806e9c0 2023-03-03T06:18:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7f97cf7fe700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f97bc068340 0x7f97bc06a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7f1b7bfff700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1b600682f0 0x7f1b6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.959 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.897+0000 7f6f517fa700 1 -- 172.21.15.90:0/3921362893 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f44051760 con 0x7f6f540c1760 2023-03-03T06:18:11.959 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.896+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6348150000 con 0x7f634806e950 2023-03-03T06:18:11.959 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.897+0000 7f633e7fc700 1 -- 172.21.15.90:0/2934999092 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f633800f050 con 0x7f634806e950 2023-03-03T06:18:11.959 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.897+0000 7f1b797fa700 1 -- 172.21.15.90:0/157004214 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b70055000 con 0x7f1b7c06cde0 2023-03-03T06:18:11.960 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.898+0000 7f633e7fc700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f632c0682f0 0x7f632c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.960 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.898+0000 7f633e7fc700 1 -- 172.21.15.90:0/2934999092 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6338021070 con 0x7f634806e950 2023-03-03T06:18:11.960 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.898+0000 7f634db6a700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f632c0682f0 0x7f632c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.960 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.898+0000 7fe09f7fe700 1 -- 172.21.15.90:0/1540038023 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe0a8019030 con 0x7fe0b806e9c0 2023-03-03T06:18:11.961 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.898+0000 7f634db6a700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f632c0682f0 0x7f632c06a7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6348068140 tx=0x7f6344009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.961 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.898+0000 7f1b7bfff700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1b600682f0 0x7f1b6006a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c1f48e0 tx=0x7f1b68006cb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.961 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.897+0000 7f97cf7fe700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f97bc068340 0x7f97bc06a7f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f97d00b4b20 tx=0x7f97c8009000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.899+0000 7fe09f7fe700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fe0a40682f0 0x7fe0a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.899+0000 7fe09f7fe700 1 -- 172.21.15.90:0/1540038023 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe0a8085cb0 con 0x7fe0b806e9c0 2023-03-03T06:18:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.899+0000 7fe0b67fc700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fe0a40682f0 0x7fe0a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.899+0000 7fe0b67fc700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fe0a40682f0 0x7fe0a406a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8066530 tx=0x7fe0ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:11.963 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.906+0000 7f97ccff9700 1 -- 172.21.15.90:0/3766344974 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97c0056440 con 0x7f97d00bfe90 2023-03-03T06:18:11.963 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.920+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b81501f0 con 0x7fe0b806e9c0 2023-03-03T06:18:11.963 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.920+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634814ccd0 con 0x7f634806e950 2023-03-03T06:18:11.963 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.925+0000 7fe09f7fe700 1 -- 172.21.15.90:0/1540038023 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe0a8017080 con 0x7fe0b806e9c0 2023-03-03T06:18:11.964 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:11.925+0000 7f633e7fc700 1 -- 172.21.15.90:0/2934999092 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6338085e60 con 0x7f634806e950 2023-03-03T06:18:12.257 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.256+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1048005590 con 0x7f10680c19d0 2023-03-03T06:18:12.258 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.257+0000 7f10657fa700 1 -- 172.21.15.90:0/1799631576 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1058055d50 con 0x7f10680c19d0 2023-03-03T06:18:12.261 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:12.262 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.260+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f10540682f0 msgr2=0x7f105406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.262 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.260+0000 7f106ed08700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f10540682f0 0x7f105406a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f105c007fd0 tx=0x7f105c007940 comp rx=0 tx=0).stop 2023-03-03T06:18:12.262 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.260+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10680c19d0 msgr2=0x7f1068071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.262 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.260+0000 7f106ed08700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10680c19d0 0x7f1068071050 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f105800e9b0 tx=0x7f105800ed70 comp rx=0 tx=0).stop 2023-03-03T06:18:12.262 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1b7c0b8650 con 0x7f1b7c06cde0 2023-03-03T06:18:12.263 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 shutdown_connections 2023-03-03T06:18:12.263 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f10540682f0 0x7f105406a7a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.263 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f106815fbe0 0x7f10680bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.263 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10680bc1f0 0x7f1068070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 --2- 172.21.15.90:0/1799631576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10680c19d0 0x7f1068071050 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 >> 172.21.15.90:0/1799631576 conn(0x7f106806afc0 msgr2=0x7f106815aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 shutdown_connections 2023-03-03T06:18:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.261+0000 7f106ed08700 1 -- 172.21.15.90:0/1799631576 wait complete. 2023-03-03T06:18:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.267+0000 7f1b797fa700 1 -- 172.21.15.90:0/157004214 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b70017020 con 0x7f1b7c06cde0 2023-03-03T06:18:12.269 INFO:teuthology.orchestra.run.smithi090.stdout:55834574853 2023-03-03T06:18:12.272 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.270+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1b600682f0 msgr2=0x7f1b6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.272 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.270+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1b600682f0 0x7f1b6006a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c1f48e0 tx=0x7f1b68006cb0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.272 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.270+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b7c06cde0 msgr2=0x7f1b7c1f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.272 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.270+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b7c06cde0 0x7f1b7c1f4bf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000c4d0 tx=0x7f1b7000c890 comp rx=0 tx=0).stop 2023-03-03T06:18:12.273 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.271+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 shutdown_connections 2023-03-03T06:18:12.273 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.271+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b7c0bb280 0x7f1b7c1f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.273 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.271+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f1b600682f0 0x7f1b6006a7a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.273 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.271+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b7c06cde0 0x7f1b7c1f4bf0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.273 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.271+0000 7f1b82e4d700 1 --2- 172.21.15.90:0/157004214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b7c0b7b80 0x7f1b7c1f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.273 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.271+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 >> 172.21.15.90:0/157004214 conn(0x7f1b7c06afc0 msgr2=0x7f1b7c0bdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.274 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.272+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 shutdown_connections 2023-03-03T06:18:12.274 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.272+0000 7f1b82e4d700 1 -- 172.21.15.90:0/157004214 wait complete. 2023-03-03T06:18:12.279 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.277+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f814404e750 con 0x7f81440bb9d0 2023-03-03T06:18:12.280 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.279+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8ec004e750 con 0x7f8ec0145510 2023-03-03T06:18:12.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-03-03T06:18:12.281 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.279+0000 7f8139ffb700 1 -- 172.21.15.90:0/2734409511 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f813c0560a0 con 0x7f81440bb9d0 2023-03-03T06:18:12.282 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.279+0000 7f8eaffff700 1 -- 172.21.15.90:0/82684619 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8eb00557f0 con 0x7f8ec0145510 2023-03-03T06:18:12.282 INFO:teuthology.orchestra.run.smithi090.stdout:55834574853 2023-03-03T06:18:12.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f8ea4068390 msgr2=0x7f8ea406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.283 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f8ea4068390 0x7f8ea406a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000f60 tx=0x7f8eb8007a40 comp rx=0 tx=0).stop 2023-03-03T06:18:12.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec0145510 msgr2=0x7f8ec0071900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec0145510 0x7f8ec0071900 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000b970 tx=0x7f8eb000bd30 comp rx=0 tx=0).stop 2023-03-03T06:18:12.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 shutdown_connections 2023-03-03T06:18:12.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f8ea4068390 0x7f8ea406a840 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec0145510 0x7f8ec0071900 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8ec00b5fd0 0x7f8ec01e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 --2- 172.21.15.90:0/82684619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ec00bd3f0 0x7f8ec01e53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.282+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 >> 172.21.15.90:0/82684619 conn(0x7f8ec006afc0 msgr2=0x7f8ec014e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.285 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.283+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 shutdown_connections 2023-03-03T06:18:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81280683c0 msgr2=0x7f812806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81280683c0 0x7f812806a870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f81340013c0 tx=0x7f8134009040 comp rx=0 tx=0).stop 2023-03-03T06:18:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81440bb9d0 msgr2=0x7f814414dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81440bb9d0 0x7f814414dea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f813c00cba0 tx=0x7f813c00ceb0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 shutdown_connections 2023-03-03T06:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81440bb9d0 0x7f814414dea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81440a6e80 0x7f81440c00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f81280683c0 0x7f812806a870 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 --2- 172.21.15.90:0/2734409511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81440bad50 0x7f814414f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 >> 172.21.15.90:0/2734409511 conn(0x7f814406afc0 msgr2=0x7f8144151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 shutdown_connections 2023-03-03T06:18:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.284+0000 7f814b61e700 1 -- 172.21.15.90:0/2734409511 wait complete. 2023-03-03T06:18:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.286+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6f5414b560 con 0x7f6f540c1760 2023-03-03T06:18:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.287+0000 7f6f517fa700 1 -- 172.21.15.90:0/3921362893 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6f44017020 con 0x7f6f540c1760 2023-03-03T06:18:12.291 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.287+0000 7f8ec603e700 1 -- 172.21.15.90:0/82684619 wait complete. 2023-03-03T06:18:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.290+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6f40068340 msgr2=0x7f6f4006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.290+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6f40068340 0x7f6f4006a7f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6f540b86d0 tx=0x7f6f48008030 comp rx=0 tx=0).stop 2023-03-03T06:18:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.290+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f540c1760 msgr2=0x7f6f54147f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.290+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f540c1760 0x7f6f54147f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400c450 tx=0x7f6f4400c760 comp rx=0 tx=0).stop 2023-03-03T06:18:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.291+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 shutdown_connections 2023-03-03T06:18:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.291+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f540c1760 0x7f6f54147f20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.291+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f6f40068340 0x7f6f4006a7f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.291+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f540b5fd0 0x7f6f540c07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.291+0000 7f6f5ae1f700 1 --2- 172.21.15.90:0/3921362893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f540bd3f0 0x7f6f541479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.291+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 >> 172.21.15.90:0/3921362893 conn(0x7f6f5406afc0 msgr2=0x7f6f540b9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.292+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 shutdown_connections 2023-03-03T06:18:12.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2023-03-03T06:18:12.298 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.297+0000 7f6f5ae1f700 1 -- 172.21.15.90:0/3921362893 wait complete. 2023-03-03T06:18:12.300 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.297+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f97b0005860 con 0x7f97d00bfe90 2023-03-03T06:18:12.301 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-03-03T06:18:12.301 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.299+0000 7f97ccff9700 1 -- 172.21.15.90:0/3766344974 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f97c0011e60 con 0x7f97d00bfe90 2023-03-03T06:18:12.302 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:12.302 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.300+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe0b80c3000 con 0x7fe0b806e9c0 2023-03-03T06:18:12.302 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.301+0000 7fe09f7fe700 1 -- 172.21.15.90:0/1540038023 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe0a8021020 con 0x7fe0b806e9c0 2023-03-03T06:18:12.302 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:12.304 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f97bc068340 msgr2=0x7f97bc06a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.305 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f97bc068340 0x7f97bc06a7f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f97d00b4b20 tx=0x7f97c8009000 comp rx=0 tx=0).stop 2023-03-03T06:18:12.305 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d00bfe90 msgr2=0x7f97d00b4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.305 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d00bfe90 0x7f97d00b4360 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f97c000b530 tx=0x7f97c000b8f0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.305 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 shutdown_connections 2023-03-03T06:18:12.305 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97d0149850 0x7f97d00a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.305 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f97bc068340 0x7f97bc06a7f0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f97d00b4b20 tx=0x7f97c8009000 comp rx=0 tx=0).stop 2023-03-03T06:18:12.305 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d00bfe90 0x7f97d00b4360 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.306 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 --2- 172.21.15.90:0/3766344974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d00c2910 0x7f97d01ecb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.306 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.303+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 >> 172.21.15.90:0/3766344974 conn(0x7f97d00c61a0 msgr2=0x7f97d00a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.306 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fe0a40682f0 msgr2=0x7fe0a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fe0a40682f0 0x7fe0a406a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8066530 tx=0x7fe0ac009040 comp rx=0 tx=0).stop 2023-03-03T06:18:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806e9c0 msgr2=0x7fe0b80a8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806e9c0 0x7fe0b80a8c00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800e9b0 tx=0x7fe0a800ed70 comp rx=0 tx=0).stop 2023-03-03T06:18:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 shutdown_connections 2023-03-03T06:18:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fe0a40682f0 0x7fe0a406a7a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b8149530 0x7fe0b80a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.308 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 shutdown_connections 2023-03-03T06:18:12.308 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7f97d6796700 1 -- 172.21.15.90:0/3766344974 wait complete. 2023-03-03T06:18:12.308 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0b806e9c0 0x7fe0b80a8c00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.308 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 --2- 172.21.15.90:0/1540038023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0b806ee30 0x7fe0b80a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.309 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.304+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 >> 172.21.15.90:0/1540038023 conn(0x7fe0b80c3270 msgr2=0x7fe0b806c910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.309 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.305+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 shutdown_connections 2023-03-03T06:18:12.309 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.305+0000 7fe0bd7da700 1 -- 172.21.15.90:0/1540038023 wait complete. 2023-03-03T06:18:12.310 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2023-03-03T06:18:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.311+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f634806f5c0 con 0x7f634806e950 2023-03-03T06:18:12.313 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.312+0000 7f633e7fc700 1 -- 172.21.15.90:0/2934999092 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f633802b070 con 0x7f634806e950 2023-03-03T06:18:12.313 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f632c0682f0 msgr2=0x7f632c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f632c0682f0 0x7f632c06a7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6348068140 tx=0x7f6344009210 comp rx=0 tx=0).stop 2023-03-03T06:18:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 msgr2=0x7f63480b70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:12.317 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f63480b70e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f63380090b0 tx=0x7f6338004750 comp rx=0 tx=0).stop 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 shutdown_connections 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f632c0682f0 0x7f632c06a7a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6348152770 0x7f6348148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f634806a330 0x7f6348150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 --2- 172.21.15.90:0/2934999092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f634806e950 0x7f63480b70e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 >> 172.21.15.90:0/2934999092 conn(0x7f634806b440 msgr2=0x7f634806bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 shutdown_connections 2023-03-03T06:18:12.318 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:12.315+0000 7f634fdce700 1 -- 172.21.15.90:0/2934999092 wait complete. 2023-03-03T06:18:12.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2023-03-03T06:18:12.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-03-03T06:18:12.325 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:12.327 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2023-03-03T06:18:12.327 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:12.331 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2023-03-03T06:18:12.332 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:13.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-03-03T06:18:13.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-03T06:18:13.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-03-03T06:18:13.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-03T06:18:13.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-03T06:18:13.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.464+0000 7f0f3b949700 1 Processor -- start 2023-03-03T06:18:13.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.464+0000 7f0f3b949700 1 -- start start 2023-03-03T06:18:13.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.465+0000 7f0f3b949700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0f341461d0 0x7f0f340beb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.465+0000 7f0f3b949700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f3414fb40 0x7f0f341567c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f3b949700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f34071a80 con 0x7f0f340bf0d0 2023-03-03T06:18:13.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f3b949700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f34070f40 con 0x7f0f341461d0 2023-03-03T06:18:13.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f3b949700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340bcda0 con 0x7f0f3414fb40 2023-03-03T06:18:13.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0f341461d0 0x7f0f340beb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0f341461d0 0x7f0f340beb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34312/0 (socket says 172.21.15.90:34312) 2023-03-03T06:18:13.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 -- 172.21.15.90:0/3964099084 learned_addr learned my addr 172.21.15.90:0/3964099084 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:13.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f39ee6700 1 --2- 172.21.15.90:0/3964099084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f3414fb40 0x7f0f341567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 -- 172.21.15.90:0/3964099084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f3414fb40 msgr2=0x7f0f341567c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:13.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 --2- 172.21.15.90:0/3964099084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f3414fb40 0x7f0f341567c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 -- 172.21.15.90:0/3964099084 >> v1:172.21.15.121:6789/0 conn(0x7f0f340bf0d0 legacy=0x7f0f3414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 -- 172.21.15.90:0/3964099084 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f341465a0 con 0x7f0f341461d0 2023-03-03T06:18:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f396e5700 1 --2- 172.21.15.90:0/3964099084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341461d0 0x7f0f340beb90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0f300098f0 tx=0x7f0f30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e790a7a9f63012 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.466+0000 7f0f2bfff700 1 -- 172.21.15.90:0/3964099084 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000b8a0 con 0x7f0f341461d0 2023-03-03T06:18:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f2bfff700 1 -- 172.21.15.90:0/3964099084 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3000ba00 con 0x7f0f341461d0 2023-03-03T06:18:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f2bfff700 1 -- 172.21.15.90:0/3964099084 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000bd90 con 0x7f0f341461d0 2023-03-03T06:18:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f3b949700 1 -- 172.21.15.90:0/3964099084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341461d0 msgr2=0x7f0f340beb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f3b949700 1 --2- 172.21.15.90:0/3964099084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341461d0 0x7f0f340beb90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0f300098f0 tx=0x7f0f30009c00 comp rx=0 tx=0).stop 2023-03-03T06:18:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f3b949700 1 -- 172.21.15.90:0/3964099084 shutdown_connections 2023-03-03T06:18:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f3b949700 1 --2- 172.21.15.90:0/3964099084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f3414fb40 0x7f0f341567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f3b949700 1 --2- 172.21.15.90:0/3964099084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341461d0 0x7f0f340beb90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.467+0000 7f0f3b949700 1 -- 172.21.15.90:0/3964099084 >> 172.21.15.90:0/3964099084 conn(0x7f0f3406b440 msgr2=0x7f0f3406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.468+0000 7f0f3b949700 1 -- 172.21.15.90:0/3964099084 shutdown_connections 2023-03-03T06:18:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.468+0000 7f0f3b949700 1 -- 172.21.15.90:0/3964099084 wait complete. 2023-03-03T06:18:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.469+0000 7f0f3b949700 1 Processor -- start 2023-03-03T06:18:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.469+0000 7f0f3b949700 1 -- start start 2023-03-03T06:18:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.470+0000 7f0f3b949700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f340bf0d0 0x7f0f341e22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.470+0000 7f0f3b949700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f341461d0 0x7f0f340bb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.470+0000 7f0f3b949700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f3414fb40 0x7f0f341d85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.470+0000 7f0f3b949700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f34148fe0 con 0x7f0f3414fb40 2023-03-03T06:18:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.470+0000 7f0f3b949700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f341e2830 con 0x7f0f340bf0d0 2023-03-03T06:18:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.470+0000 7f0f3b949700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340bbe80 con 0x7f0f341461d0 2023-03-03T06:18:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f3414fb40 0x7f0f341d85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f3414fb40 0x7f0f341d85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34322/0 (socket says 172.21.15.90:34322) 2023-03-03T06:18:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 -- 172.21.15.90:0/1050486587 learned_addr learned my addr 172.21.15.90:0/1050486587 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f396e5700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f340bf0d0 0x7f0f341e22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f38ee4700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f341461d0 0x7f0f340bb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 -- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f341461d0 msgr2=0x7f0f340bb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f341461d0 0x7f0f340bb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 -- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f340bf0d0 msgr2=0x7f0f341e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f340bf0d0 0x7f0f341e22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 -- 172.21.15.90:0/1050486587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f30009550 con 0x7f0f3414fb40 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f39ee6700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f3414fb40 0x7f0f341d85a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c007e40 tx=0x7f0f2c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f29ffb700 1 -- 172.21.15.90:0/1050486587 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00ede0 con 0x7f0f3414fb40 2023-03-03T06:18:13.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f29ffb700 1 -- 172.21.15.90:0/1050486587 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c014420 con 0x7f0f3414fb40 2023-03-03T06:18:13.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f29ffb700 1 -- 172.21.15.90:0/1050486587 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0147d0 con 0x7f0f3414fb40 2023-03-03T06:18:13.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.471+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f340722b0 con 0x7f0f3414fb40 2023-03-03T06:18:13.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.472+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f340bcae0 con 0x7f0f3414fb40 2023-03-03T06:18:13.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.473+0000 7f4642378700 1 Processor -- start 2023-03-03T06:18:13.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.473+0000 7f4642378700 1 -- start start 2023-03-03T06:18:13.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.473+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f34070770 con 0x7f0f3414fb40 2023-03-03T06:18:13.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.473+0000 7f0f29ffb700 1 -- 172.21.15.90:0/1050486587 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f2c012050 con 0x7f0f3414fb40 2023-03-03T06:18:13.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f0f29ffb700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0f180682f0 0x7f0f1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f0f29ffb700 1 -- 172.21.15.90:0/1050486587 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0f2c086310 con 0x7f0f3414fb40 2023-03-03T06:18:13.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f4642378700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f463c14f430 0x7f463c14f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f4642378700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463c071a80 con 0x7f463c1461d0 2023-03-03T06:18:13.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f4642378700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463c14fdf0 con 0x7f463c06a330 2023-03-03T06:18:13.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f4642378700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463c070f40 con 0x7f463c14f430 2023-03-03T06:18:13.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f4640915700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f463c06a330 0x7f463c06a770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:46174/0 (socket says 172.21.15.90:46174) 2023-03-03T06:18:13.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.474+0000 7f4640915700 1 -- 172.21.15.90:0/3100014252 learned_addr learned my addr 172.21.15.90:0/3100014252 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463b7fe700 1 --2- 172.21.15.90:0/3100014252 >> v2:172.21.15.121:3300/0 conn(0x7f463c14f430 0x7f463c14f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463b7fe700 1 -- 172.21.15.90:0/3100014252 >> v1:172.21.15.121:6789/0 conn(0x7f463c06a330 legacy=0x7f463c06a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463b7fe700 1 -- 172.21.15.90:0/3100014252 >> v1:172.21.15.90:6789/0 conn(0x7f463c1461d0 legacy=0x7f463c0beb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T06:18:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463b7fe700 1 -- 172.21.15.90:0/3100014252 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c0bcda0 con 0x7f463c14f430 2023-03-03T06:18:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463b7fe700 1 --2- 172.21.15.90:0/3100014252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c14f430 0x7f463c14f8b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46300097c0 tx=0x7f4630009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=345d775af718ca51 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f30320fe700 1 Processor -- start 2023-03-03T06:18:13.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f30320fe700 1 -- start start 2023-03-03T06:18:13.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463affd700 1 -- 172.21.15.90:0/3100014252 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630004230 con 0x7f463c14f430 2023-03-03T06:18:13.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463affd700 1 -- 172.21.15.90:0/3100014252 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630004390 con 0x7f463c14f430 2023-03-03T06:18:13.481 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.475+0000 7f463affd700 1 -- 172.21.15.90:0/3100014252 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300036a0 con 0x7f463c14f430 2023-03-03T06:18:13.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f30320fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f302c0c19d0 0x7f302c0b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f30320fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f302c0bc1f0 0x7f302c0b9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.482 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f30320fe700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0b7190 con 0x7f302c0bc6b0 2023-03-03T06:18:13.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f30320fe700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c0b7d80 con 0x7f302c0bc1f0 2023-03-03T06:18:13.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f30320fe700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c0b73e0 con 0x7f302c0c19d0 2023-03-03T06:18:13.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f302c0bc1f0 0x7f302c0b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f302c0bc1f0 0x7f302c0b9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34332/0 (socket says 172.21.15.90:34332) 2023-03-03T06:18:13.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 -- 172.21.15.90:0/4203413546 learned_addr learned my addr 172.21.15.90:0/4203413546 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:13.483 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302bfff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f302c0bc6b0 0x7f302c06d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:46180/0 (socket says 172.21.15.90:46180) 2023-03-03T06:18:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f4642378700 1 -- 172.21.15.90:0/3100014252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c14f430 msgr2=0x7f463c14f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f4642378700 1 --2- 172.21.15.90:0/3100014252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c14f430 0x7f463c14f8b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46300097c0 tx=0x7f4630009ad0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f4642378700 1 -- 172.21.15.90:0/3100014252 shutdown_connections 2023-03-03T06:18:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f4642378700 1 --2- 172.21.15.90:0/3100014252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c14f430 0x7f463c14f8b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f4642378700 1 -- 172.21.15.90:0/3100014252 >> 172.21.15.90:0/3100014252 conn(0x7f463c06b440 msgr2=0x7f463c06b840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 -- 172.21.15.90:0/4203413546 >> v2:172.21.15.121:3300/0 conn(0x7f302c0c19d0 msgr2=0x7f302c0b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 --2- 172.21.15.90:0/4203413546 >> v2:172.21.15.121:3300/0 conn(0x7f302c0c19d0 0x7f302c0b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 -- 172.21.15.90:0/4203413546 >> v1:172.21.15.121:6789/0 conn(0x7f302c0bc6b0 legacy=0x7f302c06d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 -- 172.21.15.90:0/4203413546 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c0bce70 con 0x7f302c0bc1f0 2023-03-03T06:18:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.476+0000 7f302affd700 1 --2- 172.21.15.90:0/4203413546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0bc1f0 0x7f302c0b9220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f302c0740f0 tx=0x7f3020009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e448f912548ff180 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7fa5f0f9d700 1 Processor -- start 2023-03-03T06:18:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7fa5f0f9d700 1 -- start start 2023-03-03T06:18:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7fa5f0f9d700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa5ec149200 0x7fa5ec0bf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7fa5f0f9d700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec0bdb40 con 0x7fa5ec0bf690 2023-03-03T06:18:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7fa5f0f9d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec0b7180 con 0x7fa5ec0bc940 2023-03-03T06:18:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7fa5f0f9d700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec0bd3c0 con 0x7fa5ec149200 2023-03-03T06:18:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7f0f396e5700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0f180682f0 0x7f0f1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.477+0000 7f4642378700 1 -- 172.21.15.90:0/3100014252 shutdown_connections 2023-03-03T06:18:13.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7f4642378700 1 -- 172.21.15.90:0/3100014252 wait complete. 2023-03-03T06:18:13.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5eb7fe700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa5ec149200 0x7fa5ec0bf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.488 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5eaffd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa5ec0bc940 0x7fa5ec14ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.90:46188/0 (socket says 172.21.15.90:46188) 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5eaffd700 1 -- 172.21.15.90:0/593394212 learned_addr learned my addr 172.21.15.90:0/593394212 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5ebfff700 1 --1- 172.21.15.90:0/593394212 >> v1:172.21.15.90:6789/0 conn(0x7fa5ec0bf690 0x7fa5ec159810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53546/0 (socket says 172.21.15.90:53546) 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5ea7fc700 1 -- 172.21.15.90:0/593394212 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282527682 0 0) 0x7fa5ec0b7180 con 0x7fa5ec0bc940 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5ea7fc700 1 -- 172.21.15.90:0/593394212 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d40035b0 con 0x7fa5ec0bc940 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5eb7fe700 1 -- 172.21.15.90:0/593394212 >> v1:172.21.15.121:6789/0 conn(0x7fa5ec0bc940 legacy=0x7fa5ec14ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5eb7fe700 1 -- 172.21.15.90:0/593394212 >> v1:172.21.15.90:6789/0 conn(0x7fa5ec0bf690 legacy=0x7fa5ec159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5eb7fe700 1 -- 172.21.15.90:0/593394212 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0b73d0 con 0x7fa5ec149200 2023-03-03T06:18:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7fa5eb7fe700 1 --2- 172.21.15.90:0/593394212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 0x7fa5ec0bf150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc009840 tx=0x7fa5dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c8dfe334a3e667 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.490 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.478+0000 7f0f396e5700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0f180682f0 0x7f0f1806a7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009c40 tx=0x7f0f30008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.479+0000 7f4642378700 1 Processor -- start 2023-03-03T06:18:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.479+0000 7f4642378700 1 -- start start 2023-03-03T06:18:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4642378700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06a330 0x7f463c0a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4642378700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463c14f430 0x7f463c13c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4642378700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c13c640 0x7f463c14dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4642378700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c136b80 con 0x7f463c06a330 2023-03-03T06:18:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4642378700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c06da10 con 0x7f463c13c640 2023-03-03T06:18:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4642378700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0a5a00 con 0x7f463c14f430 2023-03-03T06:18:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06a330 0x7f463c0a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06a330 0x7f463c0a54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34342/0 (socket says 172.21.15.90:34342) 2023-03-03T06:18:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 -- 172.21.15.90:0/912334737 learned_addr learned my addr 172.21.15.90:0/912334737 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4640915700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c13c640 0x7f463c14dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463b7fe700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463c14f430 0x7f463c13c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 -- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463c14f430 msgr2=0x7f463c13c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463c14f430 0x7f463c13c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 -- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c13c640 msgr2=0x7f463c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c13c640 0x7f463c14dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 -- 172.21.15.90:0/912334737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300094a0 con 0x7f463c06a330 2023-03-03T06:18:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463b7fe700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463c14f430 0x7f463c13c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f4640915700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c13c640 0x7f463c14dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.480+0000 7f463bfff700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06a330 0x7f463c0a54c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f463400be50 tx=0x7f463400bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7fa5ea7fc700 1 -- 172.21.15.90:0/593394212 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc011830 con 0x7fa5ec149200 2023-03-03T06:18:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f302a7fc700 1 -- 172.21.15.90:0/4203413546 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302000b870 con 0x7f302c0bc1f0 2023-03-03T06:18:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f0f29ffb700 1 -- 172.21.15.90:0/1050486587 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f2c052af0 con 0x7f0f3414fb40 2023-03-03T06:18:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f4638ff9700 1 -- 172.21.15.90:0/912334737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463400ce00 con 0x7f463c06a330 2023-03-03T06:18:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f302a7fc700 1 -- 172.21.15.90:0/4203413546 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302000b9d0 con 0x7f302c0bc1f0 2023-03-03T06:18:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f302a7fc700 1 -- 172.21.15.90:0/4203413546 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302000bd40 con 0x7f302c0bc1f0 2023-03-03T06:18:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f4638ff9700 1 -- 172.21.15.90:0/912334737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634014440 con 0x7f463c06a330 2023-03-03T06:18:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f4638ff9700 1 -- 172.21.15.90:0/912334737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46340147f0 con 0x7f463c06a330 2023-03-03T06:18:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c0bcae0 con 0x7f463c06a330 2023-03-03T06:18:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c1465a0 con 0x7f463c06a330 2023-03-03T06:18:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7fa5ea7fc700 1 -- 172.21.15.90:0/593394212 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc011990 con 0x7fa5ec149200 2023-03-03T06:18:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.482+0000 7fa5ea7fc700 1 -- 172.21.15.90:0/593394212 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc011d00 con 0x7fa5ec149200 2023-03-03T06:18:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/593394212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 msgr2=0x7fa5ec0bf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/593394212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 0x7fa5ec0bf150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc009840 tx=0x7fa5dc009b50 comp rx=0 tx=0).stop 2023-03-03T06:18:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 -- 172.21.15.90:0/4203413546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0bc1f0 msgr2=0x7f302c0b9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 --2- 172.21.15.90:0/4203413546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0bc1f0 0x7f302c0b9220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f302c0740f0 tx=0x7f3020009bd0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 -- 172.21.15.90:0/4203413546 shutdown_connections 2023-03-03T06:18:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 --2- 172.21.15.90:0/4203413546 >> v2:172.21.15.121:3300/0 conn(0x7f302c0c19d0 0x7f302c0b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 --2- 172.21.15.90:0/4203413546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0bc1f0 0x7f302c0b9220 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 -- 172.21.15.90:0/4203413546 >> 172.21.15.90:0/4203413546 conn(0x7f302c06afc0 msgr2=0x7f302c0c1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/593394212 shutdown_connections 2023-03-03T06:18:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/593394212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 0x7fa5ec0bf150 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/593394212 >> 172.21.15.90:0/593394212 conn(0x7fa5ec0c2c50 msgr2=0x7fa5ec0bea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463c0722b0 con 0x7f463c06a330 2023-03-03T06:18:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 -- 172.21.15.90:0/4203413546 shutdown_connections 2023-03-03T06:18:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7f30320fe700 1 -- 172.21.15.90:0/4203413546 wait complete. 2023-03-03T06:18:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/593394212 shutdown_connections 2023-03-03T06:18:13.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.485+0000 7f30320fe700 1 Processor -- start 2023-03-03T06:18:13.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.485+0000 7f4638ff9700 1 -- 172.21.15.90:0/912334737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f463401b030 con 0x7f463c06a330 2023-03-03T06:18:13.501 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.483+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/593394212 wait complete. 2023-03-03T06:18:13.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.485+0000 7fa5f0f9d700 1 Processor -- start 2023-03-03T06:18:13.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.485+0000 7fa5f0f9d700 1 -- start start 2023-03-03T06:18:13.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.485+0000 7f4638ff9700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f46280682f0 0x7f462806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.502 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f4638ff9700 1 -- 172.21.15.90:0/912334737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f46340864a0 con 0x7f463c06a330 2023-03-03T06:18:13.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f463b7fe700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f46280682f0 0x7f462806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7fa5f0f9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5ec0bc940 0x7fa5ec0b9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7fa5f0f9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec0bf690 0x7fa5ec1e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7fa5f0f9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 0x7fa5ec13bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.503 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7fa5f0f9d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1e4610 con 0x7fa5ec149200 2023-03-03T06:18:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7fa5f0f9d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec06dac0 con 0x7fa5ec0bf690 2023-03-03T06:18:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7fa5f0f9d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec06dda0 con 0x7fa5ec0bc940 2023-03-03T06:18:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.485+0000 7f30320fe700 1 -- start start 2023-03-03T06:18:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f30320fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f302c0bc1f0 0x7f302c1611a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f30320fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302c0bc6b0 0x7f302c155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f30320fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0c19d0 0x7f302c06f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f30320fe700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c1fdf40 con 0x7f302c0c19d0 2023-03-03T06:18:13.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f30320fe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c155ea0 con 0x7f302c0bc6b0 2023-03-03T06:18:13.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f30320fe700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c072530 con 0x7f302c0bc1f0 2023-03-03T06:18:13.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f302bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0c19d0 0x7f302c06f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f302bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0c19d0 0x7f302c06f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34356/0 (socket says 172.21.15.90:34356) 2023-03-03T06:18:13.505 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.486+0000 7f302bfff700 1 -- 172.21.15.90:0/977233625 learned_addr learned my addr 172.21.15.90:0/977233625 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:13.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f463b7fe700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f46280682f0 0x7f462806a7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4630000c00 tx=0x7f4630008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec0bf690 0x7fa5ec1e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5ec0bc940 0x7fa5ec0b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5ec0bc940 0x7fa5ec0b9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.90:39288/0 (socket says 172.21.15.90:39288) 2023-03-03T06:18:13.506 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eb7fe700 1 -- 172.21.15.90:0/934733724 learned_addr learned my addr 172.21.15.90:0/934733724 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:13.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302affd700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302c0bc6b0 0x7f302c155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302b7fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f302c0bc1f0 0x7f302c1611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.507 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302bfff700 1 -- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f302c0bc1f0 msgr2=0x7f302c1611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302bfff700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f302c0bc1f0 0x7f302c1611a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302bfff700 1 -- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302c0bc6b0 msgr2=0x7f302c155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302bfff700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302c0bc6b0 0x7f302c155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302bfff700 1 -- 172.21.15.90:0/977233625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3020009550 con 0x7f302c0c19d0 2023-03-03T06:18:13.508 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7f302bfff700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0c19d0 0x7f302c06f610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f302400cbd0 tx=0x7f302400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eaffd700 1 -- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5ec0bc940 msgr2=0x7fa5ec0b9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eaffd700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5ec0bc940 0x7fa5ec0b9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.509 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eaffd700 1 -- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 msgr2=0x7fa5ec13bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eaffd700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 0x7fa5ec13bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eaffd700 1 -- 172.21.15.90:0/934733724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc0094a0 con 0x7fa5ec0bf690 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eb7fe700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5ec0bc940 0x7fa5ec0b9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5eaffd700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec0bf690 0x7fa5ec1e4da0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c420 tx=0x7fa5e000c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.487+0000 7fa5d3fff700 1 -- 172.21.15.90:0/934733724 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0005d70 con 0x7fa5ec0bf690 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.488+0000 7fa5d3fff700 1 -- 172.21.15.90:0/934733724 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e00070d0 con 0x7fa5ec0bf690 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.488+0000 7fa5d3fff700 1 -- 172.21.15.90:0/934733724 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0007440 con 0x7fa5ec0bf690 2023-03-03T06:18:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.488+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec0b76b0 con 0x7fa5ec0bf690 2023-03-03T06:18:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.488+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec0bd050 con 0x7fa5ec0bf690 2023-03-03T06:18:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.489+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec1495d0 con 0x7fa5ec0bf690 2023-03-03T06:18:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.489+0000 7fa5d3fff700 1 -- 172.21.15.90:0/934733724 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa5e0019030 con 0x7fa5ec0bf690 2023-03-03T06:18:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.490+0000 7fa5d3fff700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa5d80682f0 0x7fa5d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.490+0000 7fa5eb7fe700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa5d80682f0 0x7fa5d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.490+0000 7fa5d3fff700 1 -- 172.21.15.90:0/934733724 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa5e0084e10 con 0x7fa5ec0bf690 2023-03-03T06:18:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.490+0000 7fa5eb7fe700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa5d80682f0 0x7fa5d806a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc000c00 tx=0x7fa5dc01c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.492+0000 7f3013fff700 1 -- 172.21.15.90:0/977233625 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302400aa60 con 0x7f302c0c19d0 2023-03-03T06:18:13.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.492+0000 7f3013fff700 1 -- 172.21.15.90:0/977233625 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302400abc0 con 0x7f302c0c19d0 2023-03-03T06:18:13.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.492+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c0bd430 con 0x7f302c0c19d0 2023-03-03T06:18:13.512 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.492+0000 7f4638ff9700 1 -- 172.21.15.90:0/912334737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4634052c60 con 0x7f463c06a330 2023-03-03T06:18:13.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.492+0000 7f3013fff700 1 -- 172.21.15.90:0/977233625 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024007070 con 0x7f302c0c19d0 2023-03-03T06:18:13.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.492+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c0bd150 con 0x7f302c0c19d0 2023-03-03T06:18:13.513 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.493+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c0bd710 con 0x7f302c0c19d0 2023-03-03T06:18:13.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.494+0000 7f3013fff700 1 -- 172.21.15.90:0/977233625 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3024019030 con 0x7f302c0c19d0 2023-03-03T06:18:13.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.494+0000 7fa5d3fff700 1 -- 172.21.15.90:0/934733724 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5e004b5e0 con 0x7fa5ec0bf690 2023-03-03T06:18:13.514 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.494+0000 7f3013fff700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3014068400 0x7f301406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.494+0000 7f3013fff700 1 -- 172.21.15.90:0/977233625 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3024085750 con 0x7f302c0c19d0 2023-03-03T06:18:13.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.498+0000 7f302b7fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3014068400 0x7f301406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.498+0000 7f302b7fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3014068400 0x7f301406a8b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f301c009970 tx=0x7f301c009110 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.515 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.502+0000 7f3013fff700 1 -- 172.21.15.90:0/977233625 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f302404b5e0 con 0x7f302c0c19d0 2023-03-03T06:18:13.525 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.524+0000 7f86502d6700 1 Processor -- start 2023-03-03T06:18:13.526 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.524+0000 7f86502d6700 1 -- start start 2023-03-03T06:18:13.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.525+0000 7f86502d6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f864814f200 0x7f864814f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.525+0000 7f86502d6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f864814fbb0 0x7f86481564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.527 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.525+0000 7f86502d6700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86480c0230 con 0x7f86480c2270 2023-03-03T06:18:13.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.525+0000 7f86502d6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86480a12f0 con 0x7f864814f200 2023-03-03T06:18:13.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.525+0000 7f86502d6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86480bc5f0 con 0x7f864814fbb0 2023-03-03T06:18:13.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e072700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f86480c2270 0x7f86480c06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53572/0 (socket says 172.21.15.90:53572) 2023-03-03T06:18:13.528 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e072700 1 -- 172.21.15.90:0/452441035 learned_addr learned my addr 172.21.15.90:0/452441035 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:13.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e873700 1 --2- 172.21.15.90:0/452441035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f864814fbb0 0x7f86481564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.529 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864d871700 1 --2- 172.21.15.90:0/452441035 >> v2:172.21.15.121:3300/0 conn(0x7f864814f200 0x7f864814f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864d070700 1 -- 172.21.15.90:0/452441035 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301794194 0 0) 0x7f86480c0230 con 0x7f86480c2270 2023-03-03T06:18:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864d070700 1 -- 172.21.15.90:0/452441035 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86340035b0 con 0x7f86480c2270 2023-03-03T06:18:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e873700 1 -- 172.21.15.90:0/452441035 >> v2:172.21.15.121:3300/0 conn(0x7f864814f200 msgr2=0x7f864814f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e873700 1 --2- 172.21.15.90:0/452441035 >> v2:172.21.15.121:3300/0 conn(0x7f864814f200 0x7f864814f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e873700 1 -- 172.21.15.90:0/452441035 >> v1:172.21.15.90:6789/0 conn(0x7f86480c2270 legacy=0x7f86480c06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e873700 1 -- 172.21.15.90:0/452441035 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864806b600 con 0x7f864814fbb0 2023-03-03T06:18:13.530 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864d871700 1 --2- 172.21.15.90:0/452441035 >> v2:172.21.15.121:3300/0 conn(0x7f864814f200 0x7f864814f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T06:18:13.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.526+0000 7f864e873700 1 --2- 172.21.15.90:0/452441035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f864814fbb0 0x7f86481564c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f864000be30 tx=0x7f864000a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfa666f42ccc0404 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.527+0000 7f864d070700 1 -- 172.21.15.90:0/452441035 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000cca0 con 0x7f864814fbb0 2023-03-03T06:18:13.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.527+0000 7f864d070700 1 -- 172.21.15.90:0/452441035 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864000ce00 con 0x7f864814fbb0 2023-03-03T06:18:13.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.527+0000 7f864d070700 1 -- 172.21.15.90:0/452441035 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640013660 con 0x7f864814fbb0 2023-03-03T06:18:13.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 -- 172.21.15.90:0/452441035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f864814fbb0 msgr2=0x7f86481564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 --2- 172.21.15.90:0/452441035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f864814fbb0 0x7f86481564c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f864000be30 tx=0x7f864000a400 comp rx=0 tx=0).stop 2023-03-03T06:18:13.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 -- 172.21.15.90:0/452441035 shutdown_connections 2023-03-03T06:18:13.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 --2- 172.21.15.90:0/452441035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f864814fbb0 0x7f86481564c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 --2- 172.21.15.90:0/452441035 >> v2:172.21.15.121:3300/0 conn(0x7f864814f200 0x7f864814f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 -- 172.21.15.90:0/452441035 >> 172.21.15.90:0/452441035 conn(0x7f86480a4180 msgr2=0x7f86480c18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 -- 172.21.15.90:0/452441035 shutdown_connections 2023-03-03T06:18:13.532 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.528+0000 7f86502d6700 1 -- 172.21.15.90:0/452441035 wait complete. 2023-03-03T06:18:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.530+0000 7f86502d6700 1 Processor -- start 2023-03-03T06:18:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.530+0000 7f86502d6700 1 -- start start 2023-03-03T06:18:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f86502d6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86480c2270 0x7f86480bcf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f86502d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f864814f200 0x7f86480be280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f86502d6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f864814fbb0 0x7f864806fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f86502d6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86480a4da0 con 0x7f864814fbb0 2023-03-03T06:18:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f86502d6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86480a5360 con 0x7f864814f200 2023-03-03T06:18:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f86502d6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86480a57f0 con 0x7f86480c2270 2023-03-03T06:18:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f864814fbb0 0x7f864806fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f864814fbb0 0x7f864806fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34376/0 (socket says 172.21.15.90:34376) 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 -- 172.21.15.90:0/483143892 learned_addr learned my addr 172.21.15.90:0/483143892 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e072700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86480c2270 0x7f86480bcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864d871700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f864814f200 0x7f86480be280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 -- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86480c2270 msgr2=0x7f86480bcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86480c2270 0x7f86480bcf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 -- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f864814f200 msgr2=0x7f86480be280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f864814f200 0x7f86480be280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.531+0000 7f864e873700 1 -- 172.21.15.90:0/483143892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863800d8d0 con 0x7f864814fbb0 2023-03-03T06:18:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.532+0000 7f864e873700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f864814fbb0 0x7f864806fde0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8640013650 tx=0x7f8640004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.532+0000 7f863effd700 1 -- 172.21.15.90:0/483143892 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000b220 con 0x7f864814fbb0 2023-03-03T06:18:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.532+0000 7f863effd700 1 -- 172.21.15.90:0/483143892 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864000b380 con 0x7f864814fbb0 2023-03-03T06:18:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.532+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864000ba90 con 0x7f864814fbb0 2023-03-03T06:18:13.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.532+0000 7f863effd700 1 -- 172.21.15.90:0/483143892 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640013c20 con 0x7f864814fbb0 2023-03-03T06:18:13.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.532+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864806b040 con 0x7f864814fbb0 2023-03-03T06:18:13.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.533+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864814ffe0 con 0x7f864814fbb0 2023-03-03T06:18:13.537 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.534+0000 7f863effd700 1 -- 172.21.15.90:0/483143892 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f864000a510 con 0x7f864814fbb0 2023-03-03T06:18:13.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.534+0000 7f863effd700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f862c0682f0 0x7f862c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:13.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.534+0000 7f863effd700 1 -- 172.21.15.90:0/483143892 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8640087dc0 con 0x7f864814fbb0 2023-03-03T06:18:13.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.535+0000 7f864e072700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f862c0682f0 0x7f862c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:13.538 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.535+0000 7f864e072700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f862c0682f0 0x7f862c06a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f86440011e0 tx=0x7f8644009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:13.544 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.543+0000 7f863effd700 1 -- 172.21.15.90:0/483143892 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f86400545a0 con 0x7f864814fbb0 2023-03-03T06:18:13.688 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.686+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0f340b8260 con 0x7f0f3414fb40 2023-03-03T06:18:13.689 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.687+0000 7f0f29ffb700 1 -- 172.21.15.90:0/1050486587 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0f2c056110 con 0x7f0f3414fb40 2023-03-03T06:18:13.689 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:13.690 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0f180682f0 msgr2=0x7f0f1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.691 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0f180682f0 0x7f0f1806a7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009c40 tx=0x7f0f30008030 comp rx=0 tx=0).stop 2023-03-03T06:18:13.691 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f3414fb40 msgr2=0x7f0f341d85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.691 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f3414fb40 0x7f0f341d85a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c007e40 tx=0x7f0f2c00e3c0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.691 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 shutdown_connections 2023-03-03T06:18:13.691 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0f180682f0 0x7f0f1806a7a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.691 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f3414fb40 0x7f0f341d85a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f340bf0d0 0x7f0f341e22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 --2- 172.21.15.90:0/1050486587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f341461d0 0x7f0f340bb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 >> 172.21.15.90:0/1050486587 conn(0x7f0f3406b440 msgr2=0x7f0f3406d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 shutdown_connections 2023-03-03T06:18:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.689+0000 7f0f3b949700 1 -- 172.21.15.90:0/1050486587 wait complete. 2023-03-03T06:18:13.697 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.696+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa5ec13ca30 con 0x7fa5ec0bf690 2023-03-03T06:18:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.696+0000 7fa5d3fff700 1 -- 172.21.15.90:0/934733724 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa5e0017020 con 0x7fa5ec0bf690 2023-03-03T06:18:13.698 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:13.700 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa5d80682f0 msgr2=0x7fa5d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.700 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa5d80682f0 0x7fa5d806a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc000c00 tx=0x7fa5dc01c040 comp rx=0 tx=0).stop 2023-03-03T06:18:13.700 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec0bf690 msgr2=0x7fa5ec1e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.700 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec0bf690 0x7fa5ec1e4da0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c420 tx=0x7fa5e000c730 comp rx=0 tx=0).stop 2023-03-03T06:18:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 shutdown_connections 2023-03-03T06:18:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fa5d80682f0 0x7fa5d806a7a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5ec149200 0x7fa5ec13bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5ec0bc940 0x7fa5ec0b9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 --2- 172.21.15.90:0/934733724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec0bf690 0x7fa5ec1e4da0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.698+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 >> 172.21.15.90:0/934733724 conn(0x7fa5ec0c2c50 msgr2=0x7fa5ec06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.699+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 shutdown_connections 2023-03-03T06:18:13.702 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.699+0000 7fa5f0f9d700 1 -- 172.21.15.90:0/934733724 wait complete. 2023-03-03T06:18:13.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-03-03T06:18:13.704 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.707+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f302c04e750 con 0x7f302c0c19d0 2023-03-03T06:18:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.707+0000 7f3013fff700 1 -- 172.21.15.90:0/977233625 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3024017090 con 0x7f302c0c19d0 2023-03-03T06:18:13.709 INFO:teuthology.orchestra.run.smithi090.stdout:55834574855 2023-03-03T06:18:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.709+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3014068400 msgr2=0x7f301406a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.709+0000 7f30320fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3014068400 0x7f301406a8b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f301c009970 tx=0x7f301c009110 comp rx=0 tx=0).stop 2023-03-03T06:18:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.709+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0c19d0 msgr2=0x7f302c06f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.709+0000 7f30320fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0c19d0 0x7f302c06f610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f302400cbd0 tx=0x7f302400cee0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 shutdown_connections 2023-03-03T06:18:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f3014068400 0x7f301406a8b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f302c0c19d0 0x7f302c06f610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f302c0bc1f0 0x7f302c1611a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 --2- 172.21.15.90:0/977233625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302c0bc6b0 0x7f302c155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 >> 172.21.15.90:0/977233625 conn(0x7f302c06afc0 msgr2=0x7f302c162070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 shutdown_connections 2023-03-03T06:18:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.710+0000 7f30320fe700 1 -- 172.21.15.90:0/977233625 wait complete. 2023-03-03T06:18:13.713 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2023-03-03T06:18:13.714 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:13.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2023-03-03T06:18:13.724 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:13.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.794+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f463c04e750 con 0x7f463c06a330 2023-03-03T06:18:13.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.794+0000 7f4638ff9700 1 -- 172.21.15.90:0/912334737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4634056280 con 0x7f463c06a330 2023-03-03T06:18:13.796 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:13.798 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f46280682f0 msgr2=0x7f462806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.798 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f46280682f0 0x7f462806a7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4630000c00 tx=0x7f4630008030 comp rx=0 tx=0).stop 2023-03-03T06:18:13.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06a330 msgr2=0x7f463c0a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06a330 0x7f463c0a54c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f463400be50 tx=0x7f463400bf30 comp rx=0 tx=0).stop 2023-03-03T06:18:13.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 shutdown_connections 2023-03-03T06:18:13.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f463c13c640 0x7f463c14dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f46280682f0 0x7f462806a7a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06a330 0x7f463c0a54c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 --2- 172.21.15.90:0/912334737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463c14f430 0x7f463c13c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 >> 172.21.15.90:0/912334737 conn(0x7f463c06b440 msgr2=0x7f463c06a770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 shutdown_connections 2023-03-03T06:18:13.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.797+0000 7f4642378700 1 -- 172.21.15.90:0/912334737 wait complete. 2023-03-03T06:18:13.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-03-03T06:18:13.812 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.841+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f864806b320 con 0x7f864814fbb0 2023-03-03T06:18:13.843 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.842+0000 7f863effd700 1 -- 172.21.15.90:0/483143892 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f864002a090 con 0x7f864814fbb0 2023-03-03T06:18:13.844 INFO:teuthology.orchestra.run.smithi090.stdout:55834574854 2023-03-03T06:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f862c0682f0 msgr2=0x7f862c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f862c0682f0 0x7f862c06a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f86440011e0 tx=0x7f8644009040 comp rx=0 tx=0).stop 2023-03-03T06:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f864814fbb0 msgr2=0x7f864806fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f864814fbb0 0x7f864806fde0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8640013650 tx=0x7f8640004ad0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 shutdown_connections 2023-03-03T06:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f862c0682f0 0x7f862c06a7a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f864814fbb0 0x7f864806fde0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.846 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86480c2270 0x7f86480bcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 --2- 172.21.15.90:0/483143892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f864814f200 0x7f86480be280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 >> 172.21.15.90:0/483143892 conn(0x7f86480a4180 msgr2=0x7f864806e540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 shutdown_connections 2023-03-03T06:18:13.847 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:13.844+0000 7f86502d6700 1 -- 172.21.15.90:0/483143892 wait complete. 2023-03-03T06:18:13.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2023-03-03T06:18:13.857 DEBUG:teuthology.parallel:result is None 2023-03-03T06:18:13.858 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-03-03T06:18:13.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T06:18:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.015+0000 7fce62030700 1 Processor -- start 2023-03-03T06:18:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.015+0000 7fce62030700 1 -- start start 2023-03-03T06:18:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce62030700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fce5c14f7d0 0x7fce5c15aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce62030700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5c153340 con 0x7fce5c06ea90 2023-03-03T06:18:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce62030700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5c1497f0 con 0x7fce5c149420 2023-03-03T06:18:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce62030700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5c152750 con 0x7fce5c14f7d0 2023-03-03T06:18:14.019 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5bfff700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fce5c14f7d0 0x7fce5c15aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5bfff700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fce5c14f7d0 0x7fce5c15aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34384/0 (socket says 172.21.15.90:34384) 2023-03-03T06:18:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5affd700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fce5c06ea90 0x7fce5c14f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53588/0 (socket says 172.21.15.90:53588) 2023-03-03T06:18:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5bfff700 1 -- 172.21.15.90:0/1982913014 learned_addr learned my addr 172.21.15.90:0/1982913014 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5bfff700 1 -- 172.21.15.90:0/1982913014 >> v1:172.21.15.121:6789/0 conn(0x7fce5c149420 legacy=0x7fce5c0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.020 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5bfff700 1 -- 172.21.15.90:0/1982913014 >> v1:172.21.15.90:6789/0 conn(0x7fce5c06ea90 legacy=0x7fce5c14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5bfff700 1 -- 172.21.15.90:0/1982913014 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c152a10 con 0x7fce5c14f7d0 2023-03-03T06:18:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5bfff700 1 --2- 172.21.15.90:0/1982913014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c14f7d0 0x7fce5c15aa50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fce54009e50 tx=0x7fce5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f442fbeee40dfa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5a7fc700 1 -- 172.21.15.90:0/1982913014 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400e070 con 0x7fce5c14f7d0 2023-03-03T06:18:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5a7fc700 1 -- 172.21.15.90:0/1982913014 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5400b9c0 con 0x7fce5c14f7d0 2023-03-03T06:18:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.016+0000 7fce5a7fc700 1 -- 172.21.15.90:0/1982913014 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400bd70 con 0x7fce5c14f7d0 2023-03-03T06:18:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.017+0000 7fce62030700 1 -- 172.21.15.90:0/1982913014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c14f7d0 msgr2=0x7fce5c15aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.017+0000 7fce62030700 1 --2- 172.21.15.90:0/1982913014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c14f7d0 0x7fce5c15aa50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fce54009e50 tx=0x7fce5400b330 comp rx=0 tx=0).stop 2023-03-03T06:18:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.017+0000 7fce62030700 1 -- 172.21.15.90:0/1982913014 shutdown_connections 2023-03-03T06:18:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.017+0000 7fce62030700 1 --2- 172.21.15.90:0/1982913014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c14f7d0 0x7fce5c15aa50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.017+0000 7fce62030700 1 -- 172.21.15.90:0/1982913014 >> 172.21.15.90:0/1982913014 conn(0x7fce5c06c0d0 msgr2=0x7fce5c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.017+0000 7fce62030700 1 -- 172.21.15.90:0/1982913014 shutdown_connections 2023-03-03T06:18:14.022 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.017+0000 7fce62030700 1 -- 172.21.15.90:0/1982913014 wait complete. 2023-03-03T06:18:14.024 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.018+0000 7fce62030700 1 Processor -- start 2023-03-03T06:18:14.024 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.018+0000 7fce62030700 1 -- start start 2023-03-03T06:18:14.024 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce62030700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce5c06ea90 0x7fce5c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce62030700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c149420 0x7fce5c1e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce62030700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce5c14f7d0 0x7fce5c1e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce62030700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c14d780 con 0x7fce5c149420 2023-03-03T06:18:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce62030700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c1434a0 con 0x7fce5c14f7d0 2023-03-03T06:18:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce62030700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c06aae0 con 0x7fce5c06ea90 2023-03-03T06:18:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c149420 0x7fce5c1e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c149420 0x7fce5c1e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34390/0 (socket says 172.21.15.90:34390) 2023-03-03T06:18:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 -- 172.21.15.90:0/645994446 learned_addr learned my addr 172.21.15.90:0/645994446 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5bfff700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce5c14f7d0 0x7fce5c1e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5b7fe700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce5c06ea90 0x7fce5c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 -- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce5c06ea90 msgr2=0x7fce5c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce5c06ea90 0x7fce5c06a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 -- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce5c14f7d0 msgr2=0x7fce5c1e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.026 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce5c14f7d0 0x7fce5c1e8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 -- 172.21.15.90:0/645994446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54009b30 con 0x7fce5c149420 2023-03-03T06:18:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.019+0000 7fce5affd700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c149420 0x7fce5c1e4d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00c530 tx=0x7fce4c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.020+0000 7fce43fff700 1 -- 172.21.15.90:0/645994446 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00cea0 con 0x7fce5c149420 2023-03-03T06:18:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.020+0000 7fce43fff700 1 -- 172.21.15.90:0/645994446 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c005cb0 con 0x7fce5c149420 2023-03-03T06:18:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.020+0000 7fce43fff700 1 -- 172.21.15.90:0/645994446 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c0072d0 con 0x7fce5c149420 2023-03-03T06:18:14.027 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.020+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5c152fd0 con 0x7fce5c149420 2023-03-03T06:18:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.020+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c152cf0 con 0x7fce5c149420 2023-03-03T06:18:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.020+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5c06e610 con 0x7fce5c149420 2023-03-03T06:18:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.021+0000 7fce43fff700 1 -- 172.21.15.90:0/645994446 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fce4c019030 con 0x7fce5c149420 2023-03-03T06:18:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.021+0000 7fce43fff700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fce480682f0 0x7fce4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.021+0000 7fce43fff700 1 -- 172.21.15.90:0/645994446 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fce4c084d00 con 0x7fce5c149420 2023-03-03T06:18:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.021+0000 7fce5b7fe700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fce480682f0 0x7fce4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.022+0000 7fce5b7fe700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fce480682f0 0x7fce4806a7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce50007fd0 tx=0x7fce50009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.029 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.025+0000 7fce43fff700 1 -- 172.21.15.90:0/645994446 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce4c04b5e0 con 0x7fce5c149420 2023-03-03T06:18:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.203+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 --> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce5c04e750 con 0x7fce480682f0 2023-03-03T06:18:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.211+0000 7fce43fff700 1 -- 172.21.15.90:0/645994446 <== mgr.4108 v2:172.21.15.90:6832/96475 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286476 (secure 0 0 0) 0x7fce5c04e750 con 0x7fce480682f0 2023-03-03T06:18:14.216 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:14.218 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fce480682f0 msgr2=0x7fce4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fce480682f0 0x7fce4806a7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce50007fd0 tx=0x7fce50009040 comp rx=0 tx=0).stop 2023-03-03T06:18:14.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c149420 msgr2=0x7fce5c1e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c149420 0x7fce5c1e4d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00c530 tx=0x7fce4c00c840 comp rx=0 tx=0).stop 2023-03-03T06:18:14.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 shutdown_connections 2023-03-03T06:18:14.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce5c14f7d0 0x7fce5c1e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.219 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce5c06ea90 0x7fce5c06a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fce480682f0 0x7fce4806a7a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 --2- 172.21.15.90:0/645994446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce5c149420 0x7fce5c1e4d90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 >> 172.21.15.90:0/645994446 conn(0x7fce5c06c0d0 msgr2=0x7fce5c0bee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:14.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 shutdown_connections 2023-03-03T06:18:14.220 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.214+0000 7fce62030700 1 -- 172.21.15.90:0/645994446 wait complete. 2023-03-03T06:18:14.221 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-03-03T06:18:14.227 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-03-03T06:18:14.078244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66280,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":754908440,"statfs":{"total":773094113280,"available":773026242560,"internally_reserved":0,"allocated":3186688,"data_stored":1902188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.323681"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089798+0000","last_change":"2023-03-03T06:18:05.768223+0000","last_active":"2023-03-03T06:18:07.089798+0000","last_peered":"2023-03-03T06:18:07.089798+0000","last_clean":"2023-03-03T06:18:07.089798+0000","last_became_active":"2023-03-03T06:18:05.767943+0000","last_became_peered":"2023-03-03T06:18:05.767943+0000","last_unstale":"2023-03-03T06:18:07.089798+0000","last_undegraded":"2023-03-03T06:18:07.089798+0000","last_fullsized":"2023-03-03T06:18:07.089798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:38.546721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767372+0000","last_change":"2023-03-03T06:18:05.779938+0000","last_active":"2023-03-03T06:18:06.767372+0000","last_peered":"2023-03-03T06:18:06.767372+0000","last_clean":"2023-03-03T06:18:06.767372+0000","last_became_active":"2023-03-03T06:18:05.779669+0000","last_became_peered":"2023-03-03T06:18:05.779669+0000","last_unstale":"2023-03-03T06:18:06.767372+0000","last_undegraded":"2023-03-03T06:18:06.767372+0000","last_fullsized":"2023-03-03T06:18:06.767372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088222+0000","last_change":"2023-03-03T06:18:05.773328+0000","last_active":"2023-03-03T06:18:07.088222+0000","last_peered":"2023-03-03T06:18:07.088222+0000","last_clean":"2023-03-03T06:18:07.088222+0000","last_became_active":"2023-03-03T06:18:05.773065+0000","last_became_peered":"2023-03-03T06:18:05.773065+0000","last_unstale":"2023-03-03T06:18:07.088222+0000","last_undegraded":"2023-03-03T06:18:07.088222+0000","last_fullsized":"2023-03-03T06:18:07.088222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:58.062737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089819+0000","last_change":"2023-03-03T06:18:05.773425+0000","last_active":"2023-03-03T06:18:07.089819+0000","last_peered":"2023-03-03T06:18:07.089819+0000","last_clean":"2023-03-03T06:18:07.089819+0000","last_became_active":"2023-03-03T06:18:05.772997+0000","last_became_peered":"2023-03-03T06:18:05.772997+0000","last_unstale":"2023-03-03T06:18:07.089819+0000","last_undegraded":"2023-03-03T06:18:07.089819+0000","last_fullsized":"2023-03-03T06:18:07.089819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:43.144401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087652+0000","last_change":"2023-03-03T06:18:05.771575+0000","last_active":"2023-03-03T06:18:07.087652+0000","last_peered":"2023-03-03T06:18:07.087652+0000","last_clean":"2023-03-03T06:18:07.087652+0000","last_became_active":"2023-03-03T06:18:05.771326+0000","last_became_peered":"2023-03-03T06:18:05.771326+0000","last_unstale":"2023-03-03T06:18:07.087652+0000","last_undegraded":"2023-03-03T06:18:07.087652+0000","last_fullsized":"2023-03-03T06:18:07.087652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088086+0000","last_change":"2023-03-03T06:18:05.773873+0000","last_active":"2023-03-03T06:18:07.088086+0000","last_peered":"2023-03-03T06:18:07.088086+0000","last_clean":"2023-03-03T06:18:07.088086+0000","last_became_active":"2023-03-03T06:18:05.773610+0000","last_became_peered":"2023-03-03T06:18:05.773610+0000","last_unstale":"2023-03-03T06:18:07.088086+0000","last_undegraded":"2023-03-03T06:18:07.088086+0000","last_fullsized":"2023-03-03T06:18:07.088086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:24:21.809616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766274+0000","last_change":"2023-03-03T06:18:05.779587+0000","last_active":"2023-03-03T06:18:06.766274+0000","last_peered":"2023-03-03T06:18:06.766274+0000","last_clean":"2023-03-03T06:18:06.766274+0000","last_became_active":"2023-03-03T06:18:05.779147+0000","last_became_peered":"2023-03-03T06:18:05.779147+0000","last_unstale":"2023-03-03T06:18:06.766274+0000","last_undegraded":"2023-03-03T06:18:06.766274+0000","last_fullsized":"2023-03-03T06:18:06.766274+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766529+0000","last_change":"2023-03-03T06:18:05.774633+0000","last_active":"2023-03-03T06:18:06.766529+0000","last_peered":"2023-03-03T06:18:06.766529+0000","last_clean":"2023-03-03T06:18:06.766529+0000","last_became_active":"2023-03-03T06:18:05.773725+0000","last_became_peered":"2023-03-03T06:18:05.773725+0000","last_unstale":"2023-03-03T06:18:06.766529+0000","last_undegraded":"2023-03-03T06:18:06.766529+0000","last_fullsized":"2023-03-03T06:18:06.766529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087720+0000","last_change":"2023-03-03T06:18:05.770503+0000","last_active":"2023-03-03T06:18:07.087720+0000","last_peered":"2023-03-03T06:18:07.087720+0000","last_clean":"2023-03-03T06:18:07.087720+0000","last_became_active":"2023-03-03T06:18:05.770093+0000","last_became_peered":"2023-03-03T06:18:05.770093+0000","last_unstale":"2023-03-03T06:18:07.087720+0000","last_undegraded":"2023-03-03T06:18:07.087720+0000","last_fullsized":"2023-03-03T06:18:07.087720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767476+0000","last_change":"2023-03-03T06:18:05.774425+0000","last_active":"2023-03-03T06:18:06.767476+0000","last_peered":"2023-03-03T06:18:06.767476+0000","last_clean":"2023-03-03T06:18:06.767476+0000","last_became_active":"2023-03-03T06:18:05.774137+0000","last_became_peered":"2023-03-03T06:18:05.774137+0000","last_unstale":"2023-03-03T06:18:06.767476+0000","last_undegraded":"2023-03-03T06:18:06.767476+0000","last_fullsized":"2023-03-03T06:18:06.767476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088236+0000","last_change":"2023-03-03T06:18:05.779203+0000","last_active":"2023-03-03T06:18:07.088236+0000","last_peered":"2023-03-03T06:18:07.088236+0000","last_clean":"2023-03-03T06:18:07.088236+0000","last_became_active":"2023-03-03T06:18:05.778887+0000","last_became_peered":"2023-03-03T06:18:05.778887+0000","last_unstale":"2023-03-03T06:18:07.088236+0000","last_undegraded":"2023-03-03T06:18:07.088236+0000","last_fullsized":"2023-03-03T06:18:07.088236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089765+0000","last_change":"2023-03-03T06:18:05.771290+0000","last_active":"2023-03-03T06:18:07.089765+0000","last_peered":"2023-03-03T06:18:07.089765+0000","last_clean":"2023-03-03T06:18:07.089765+0000","last_became_active":"2023-03-03T06:18:05.771073+0000","last_became_peered":"2023-03-03T06:18:05.771073+0000","last_unstale":"2023-03-03T06:18:07.089765+0000","last_undegraded":"2023-03-03T06:18:07.089765+0000","last_fullsized":"2023-03-03T06:18:07.089765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:01.951025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089750+0000","last_change":"2023-03-03T06:18:05.772434+0000","last_active":"2023-03-03T06:18:07.089750+0000","last_peered":"2023-03-03T06:18:07.089750+0000","last_clean":"2023-03-03T06:18:07.089750+0000","last_became_active":"2023-03-03T06:18:05.772171+0000","last_became_peered":"2023-03-03T06:18:05.772171+0000","last_unstale":"2023-03-03T06:18:07.089750+0000","last_undegraded":"2023-03-03T06:18:07.089750+0000","last_fullsized":"2023-03-03T06:18:07.089750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:43.200309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766717+0000","last_change":"2023-03-03T06:18:05.775751+0000","last_active":"2023-03-03T06:18:06.766717+0000","last_peered":"2023-03-03T06:18:06.766717+0000","last_clean":"2023-03-03T06:18:06.766717+0000","last_became_active":"2023-03-03T06:18:05.775268+0000","last_became_peered":"2023-03-03T06:18:05.775268+0000","last_unstale":"2023-03-03T06:18:06.766717+0000","last_undegraded":"2023-03-03T06:18:06.766717+0000","last_fullsized":"2023-03-03T06:18:06.766717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766405+0000","last_change":"2023-03-03T06:18:05.779415+0000","last_active":"2023-03-03T06:18:06.766405+0000","last_peered":"2023-03-03T06:18:06.766405+0000","last_clean":"2023-03-03T06:18:06.766405+0000","last_became_active":"2023-03-03T06:18:05.778767+0000","last_became_peered":"2023-03-03T06:18:05.778767+0000","last_unstale":"2023-03-03T06:18:06.766405+0000","last_undegraded":"2023-03-03T06:18:06.766405+0000","last_fullsized":"2023-03-03T06:18:06.766405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088011+0000","last_change":"2023-03-03T06:18:05.768282+0000","last_active":"2023-03-03T06:18:07.088011+0000","last_peered":"2023-03-03T06:18:07.088011+0000","last_clean":"2023-03-03T06:18:07.088011+0000","last_became_active":"2023-03-03T06:18:05.767972+0000","last_became_peered":"2023-03-03T06:18:05.767972+0000","last_unstale":"2023-03-03T06:18:07.088011+0000","last_undegraded":"2023-03-03T06:18:07.088011+0000","last_fullsized":"2023-03-03T06:18:07.088011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:46.885266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087730+0000","last_change":"2023-03-03T06:18:05.770349+0000","last_active":"2023-03-03T06:18:07.087730+0000","last_peered":"2023-03-03T06:18:07.087730+0000","last_clean":"2023-03-03T06:18:07.087730+0000","last_became_active":"2023-03-03T06:18:05.769801+0000","last_became_peered":"2023-03-03T06:18:05.769801+0000","last_unstale":"2023-03-03T06:18:07.087730+0000","last_undegraded":"2023-03-03T06:18:07.087730+0000","last_fullsized":"2023-03-03T06:18:07.087730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088273+0000","last_change":"2023-03-03T06:18:05.769983+0000","last_active":"2023-03-03T06:18:07.088273+0000","last_peered":"2023-03-03T06:18:07.088273+0000","last_clean":"2023-03-03T06:18:07.088273+0000","last_became_active":"2023-03-03T06:18:05.769713+0000","last_became_peered":"2023-03-03T06:18:05.769713+0000","last_unstale":"2023-03-03T06:18:07.088273+0000","last_undegraded":"2023-03-03T06:18:07.088273+0000","last_fullsized":"2023-03-03T06:18:07.088273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:48.612674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089788+0000","last_change":"2023-03-03T06:18:05.772674+0000","last_active":"2023-03-03T06:18:07.089788+0000","last_peered":"2023-03-03T06:18:07.089788+0000","last_clean":"2023-03-03T06:18:07.089788+0000","last_became_active":"2023-03-03T06:18:05.772420+0000","last_became_peered":"2023-03-03T06:18:05.772420+0000","last_unstale":"2023-03-03T06:18:07.089788+0000","last_undegraded":"2023-03-03T06:18:07.089788+0000","last_fullsized":"2023-03-03T06:18:07.089788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:17.201682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088180+0000","last_change":"2023-03-03T06:18:05.779465+0000","last_active":"2023-03-03T06:18:07.088180+0000","last_peered":"2023-03-03T06:18:07.088180+0000","last_clean":"2023-03-03T06:18:07.088180+0000","last_became_active":"2023-03-03T06:18:05.779177+0000","last_became_peered":"2023-03-03T06:18:05.779177+0000","last_unstale":"2023-03-03T06:18:07.088180+0000","last_undegraded":"2023-03-03T06:18:07.088180+0000","last_fullsized":"2023-03-03T06:18:07.088180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766718+0000","last_change":"2023-03-03T06:18:05.769339+0000","last_active":"2023-03-03T06:18:06.766718+0000","last_peered":"2023-03-03T06:18:06.766718+0000","last_clean":"2023-03-03T06:18:06.766718+0000","last_became_active":"2023-03-03T06:18:05.768277+0000","last_became_peered":"2023-03-03T06:18:05.768277+0000","last_unstale":"2023-03-03T06:18:06.766718+0000","last_undegraded":"2023-03-03T06:18:06.766718+0000","last_fullsized":"2023-03-03T06:18:06.766718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087614+0000","last_change":"2023-03-03T06:18:05.780097+0000","last_active":"2023-03-03T06:18:07.087614+0000","last_peered":"2023-03-03T06:18:07.087614+0000","last_clean":"2023-03-03T06:18:07.087614+0000","last_became_active":"2023-03-03T06:18:05.779838+0000","last_became_peered":"2023-03-03T06:18:05.779838+0000","last_unstale":"2023-03-03T06:18:07.087614+0000","last_undegraded":"2023-03-03T06:18:07.087614+0000","last_fullsized":"2023-03-03T06:18:07.087614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761222+0000","last_change":"2023-03-03T06:18:05.772869+0000","last_active":"2023-03-03T06:18:06.761222+0000","last_peered":"2023-03-03T06:18:06.761222+0000","last_clean":"2023-03-03T06:18:06.761222+0000","last_became_active":"2023-03-03T06:18:05.772498+0000","last_became_peered":"2023-03-03T06:18:05.772498+0000","last_unstale":"2023-03-03T06:18:06.761222+0000","last_undegraded":"2023-03-03T06:18:06.761222+0000","last_fullsized":"2023-03-03T06:18:06.761222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766506+0000","last_change":"2023-03-03T06:18:05.769326+0000","last_active":"2023-03-03T06:18:06.766506+0000","last_peered":"2023-03-03T06:18:06.766506+0000","last_clean":"2023-03-03T06:18:06.766506+0000","last_became_active":"2023-03-03T06:18:05.768781+0000","last_became_peered":"2023-03-03T06:18:05.768781+0000","last_unstale":"2023-03-03T06:18:06.766506+0000","last_undegraded":"2023-03-03T06:18:06.766506+0000","last_fullsized":"2023-03-03T06:18:06.766506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086250+0000","last_change":"2023-03-03T06:18:05.777824+0000","last_active":"2023-03-03T06:18:07.086250+0000","last_peered":"2023-03-03T06:18:07.086250+0000","last_clean":"2023-03-03T06:18:07.086250+0000","last_became_active":"2023-03-03T06:18:05.777221+0000","last_became_peered":"2023-03-03T06:18:05.777221+0000","last_unstale":"2023-03-03T06:18:07.086250+0000","last_undegraded":"2023-03-03T06:18:07.086250+0000","last_fullsized":"2023-03-03T06:18:07.086250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086239+0000","last_change":"2023-03-03T06:18:05.777434+0000","last_active":"2023-03-03T06:18:07.086239+0000","last_peered":"2023-03-03T06:18:07.086239+0000","last_clean":"2023-03-03T06:18:07.086239+0000","last_became_active":"2023-03-03T06:18:05.776857+0000","last_became_peered":"2023-03-03T06:18:05.776857+0000","last_unstale":"2023-03-03T06:18:07.086239+0000","last_undegraded":"2023-03-03T06:18:07.086239+0000","last_fullsized":"2023-03-03T06:18:07.086239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767237+0000","last_change":"2023-03-03T06:18:05.776802+0000","last_active":"2023-03-03T06:18:06.767237+0000","last_peered":"2023-03-03T06:18:06.767237+0000","last_clean":"2023-03-03T06:18:06.767237+0000","last_became_active":"2023-03-03T06:18:05.776531+0000","last_became_peered":"2023-03-03T06:18:05.776531+0000","last_unstale":"2023-03-03T06:18:06.767237+0000","last_undegraded":"2023-03-03T06:18:06.767237+0000","last_fullsized":"2023-03-03T06:18:06.767237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088118+0000","last_change":"2023-03-03T06:18:05.374916+0000","last_active":"2023-03-03T06:18:07.088118+0000","last_peered":"2023-03-03T06:18:07.088118+0000","last_clean":"2023-03-03T06:18:07.088118+0000","last_became_active":"2023-03-03T06:18:05.374663+0000","last_became_peered":"2023-03-03T06:18:05.374663+0000","last_unstale":"2023-03-03T06:18:07.088118+0000","last_undegraded":"2023-03-03T06:18:07.088118+0000","last_fullsized":"2023-03-03T06:18:07.088118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:19:22.426709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761867+0000","last_change":"2023-03-03T06:18:05.778939+0000","last_active":"2023-03-03T06:18:06.761867+0000","last_peered":"2023-03-03T06:18:06.761867+0000","last_clean":"2023-03-03T06:18:06.761867+0000","last_became_active":"2023-03-03T06:18:05.778350+0000","last_became_peered":"2023-03-03T06:18:05.778350+0000","last_unstale":"2023-03-03T06:18:06.761867+0000","last_undegraded":"2023-03-03T06:18:06.761867+0000","last_fullsized":"2023-03-03T06:18:06.761867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086156+0000","last_change":"2023-03-03T06:18:05.369187+0000","last_active":"2023-03-03T06:18:07.086156+0000","last_peered":"2023-03-03T06:18:07.086156+0000","last_clean":"2023-03-03T06:18:07.086156+0000","last_became_active":"2023-03-03T06:18:05.368706+0000","last_became_peered":"2023-03-03T06:18:05.368706+0000","last_unstale":"2023-03-03T06:18:07.086156+0000","last_undegraded":"2023-03-03T06:18:07.086156+0000","last_fullsized":"2023-03-03T06:18:07.086156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087690+0000","last_change":"2023-03-03T06:18:05.771810+0000","last_active":"2023-03-03T06:18:07.087690+0000","last_peered":"2023-03-03T06:18:07.087690+0000","last_clean":"2023-03-03T06:18:07.087690+0000","last_became_active":"2023-03-03T06:18:05.771484+0000","last_became_peered":"2023-03-03T06:18:05.771484+0000","last_unstale":"2023-03-03T06:18:07.087690+0000","last_undegraded":"2023-03-03T06:18:07.087690+0000","last_fullsized":"2023-03-03T06:18:07.087690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761169+0000","last_change":"2023-03-03T06:18:05.378838+0000","last_active":"2023-03-03T06:18:06.761169+0000","last_peered":"2023-03-03T06:18:06.761169+0000","last_clean":"2023-03-03T06:18:06.761169+0000","last_became_active":"2023-03-03T06:18:05.378525+0000","last_became_peered":"2023-03-03T06:18:05.378525+0000","last_unstale":"2023-03-03T06:18:06.761169+0000","last_undegraded":"2023-03-03T06:18:06.761169+0000","last_fullsized":"2023-03-03T06:18:06.761169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089855+0000","last_change":"2023-03-03T06:18:05.776315+0000","last_active":"2023-03-03T06:18:07.089855+0000","last_peered":"2023-03-03T06:18:07.089855+0000","last_clean":"2023-03-03T06:18:07.089855+0000","last_became_active":"2023-03-03T06:18:05.775728+0000","last_became_peered":"2023-03-03T06:18:05.775728+0000","last_unstale":"2023-03-03T06:18:07.089855+0000","last_undegraded":"2023-03-03T06:18:07.089855+0000","last_fullsized":"2023-03-03T06:18:07.089855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:17.287057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087740+0000","last_change":"2023-03-03T06:18:05.378086+0000","last_active":"2023-03-03T06:18:07.087740+0000","last_peered":"2023-03-03T06:18:07.087740+0000","last_clean":"2023-03-03T06:18:07.087740+0000","last_became_active":"2023-03-03T06:18:05.377265+0000","last_became_peered":"2023-03-03T06:18:05.377265+0000","last_unstale":"2023-03-03T06:18:07.087740+0000","last_undegraded":"2023-03-03T06:18:07.087740+0000","last_fullsized":"2023-03-03T06:18:07.087740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761128+0000","last_change":"2023-03-03T06:18:05.778565+0000","last_active":"2023-03-03T06:18:06.761128+0000","last_peered":"2023-03-03T06:18:06.761128+0000","last_clean":"2023-03-03T06:18:06.761128+0000","last_became_active":"2023-03-03T06:18:05.778231+0000","last_became_peered":"2023-03-03T06:18:05.778231+0000","last_unstale":"2023-03-03T06:18:06.761128+0000","last_undegraded":"2023-03-03T06:18:06.761128+0000","last_fullsized":"2023-03-03T06:18:06.761128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767346+0000","last_change":"2023-03-03T06:18:05.370256+0000","last_active":"2023-03-03T06:18:06.767346+0000","last_peered":"2023-03-03T06:18:06.767346+0000","last_clean":"2023-03-03T06:18:06.767346+0000","last_became_active":"2023-03-03T06:18:05.369970+0000","last_became_peered":"2023-03-03T06:18:05.369970+0000","last_unstale":"2023-03-03T06:18:06.767346+0000","last_undegraded":"2023-03-03T06:18:06.767346+0000","last_fullsized":"2023-03-03T06:18:06.767346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761149+0000","last_change":"2023-03-03T06:18:05.766019+0000","last_active":"2023-03-03T06:18:06.761149+0000","last_peered":"2023-03-03T06:18:06.761149+0000","last_clean":"2023-03-03T06:18:06.761149+0000","last_became_active":"2023-03-03T06:18:05.765768+0000","last_became_peered":"2023-03-03T06:18:05.765768+0000","last_unstale":"2023-03-03T06:18:06.761149+0000","last_undegraded":"2023-03-03T06:18:06.761149+0000","last_fullsized":"2023-03-03T06:18:06.761149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766991+0000","last_change":"2023-03-03T06:18:05.367491+0000","last_active":"2023-03-03T06:18:06.766991+0000","last_peered":"2023-03-03T06:18:06.766991+0000","last_clean":"2023-03-03T06:18:06.766991+0000","last_became_active":"2023-03-03T06:18:05.367177+0000","last_became_peered":"2023-03-03T06:18:05.367177+0000","last_unstale":"2023-03-03T06:18:06.766991+0000","last_undegraded":"2023-03-03T06:18:06.766991+0000","last_fullsized":"2023-03-03T06:18:06.766991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:55.267658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086167+0000","last_change":"2023-03-03T06:18:05.778585+0000","last_active":"2023-03-03T06:18:07.086167+0000","last_peered":"2023-03-03T06:18:07.086167+0000","last_clean":"2023-03-03T06:18:07.086167+0000","last_became_active":"2023-03-03T06:18:05.777053+0000","last_became_peered":"2023-03-03T06:18:05.777053+0000","last_unstale":"2023-03-03T06:18:07.086167+0000","last_undegraded":"2023-03-03T06:18:07.086167+0000","last_fullsized":"2023-03-03T06:18:07.086167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088203+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:07.088203+0000","last_peered":"2023-03-03T06:18:07.088203+0000","last_clean":"2023-03-03T06:18:07.088203+0000","last_became_active":"2023-03-03T06:18:05.372051+0000","last_became_peered":"2023-03-03T06:18:05.372051+0000","last_unstale":"2023-03-03T06:18:07.088203+0000","last_undegraded":"2023-03-03T06:18:07.088203+0000","last_fullsized":"2023-03-03T06:18:07.088203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088167+0000","last_change":"2023-03-03T06:18:05.779300+0000","last_active":"2023-03-03T06:18:07.088167+0000","last_peered":"2023-03-03T06:18:07.088167+0000","last_clean":"2023-03-03T06:18:07.088167+0000","last_became_active":"2023-03-03T06:18:05.778878+0000","last_became_peered":"2023-03-03T06:18:05.778878+0000","last_unstale":"2023-03-03T06:18:07.088167+0000","last_undegraded":"2023-03-03T06:18:07.088167+0000","last_fullsized":"2023-03-03T06:18:07.088167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766613+0000","last_change":"2023-03-03T06:18:05.377498+0000","last_active":"2023-03-03T06:18:06.766613+0000","last_peered":"2023-03-03T06:18:06.766613+0000","last_clean":"2023-03-03T06:18:06.766613+0000","last_became_active":"2023-03-03T06:18:05.377168+0000","last_became_peered":"2023-03-03T06:18:05.377168+0000","last_unstale":"2023-03-03T06:18:06.766613+0000","last_undegraded":"2023-03-03T06:18:06.766613+0000","last_fullsized":"2023-03-03T06:18:06.766613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:38.382151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761390+0000","last_change":"2023-03-03T06:18:05.779094+0000","last_active":"2023-03-03T06:18:06.761390+0000","last_peered":"2023-03-03T06:18:06.761390+0000","last_clean":"2023-03-03T06:18:06.761390+0000","last_became_active":"2023-03-03T06:18:05.778802+0000","last_became_peered":"2023-03-03T06:18:05.778802+0000","last_unstale":"2023-03-03T06:18:06.761390+0000","last_undegraded":"2023-03-03T06:18:06.761390+0000","last_fullsized":"2023-03-03T06:18:06.761390+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087593+0000","last_change":"2023-03-03T06:18:05.376181+0000","last_active":"2023-03-03T06:18:07.087593+0000","last_peered":"2023-03-03T06:18:07.087593+0000","last_clean":"2023-03-03T06:18:07.087593+0000","last_became_active":"2023-03-03T06:18:05.375857+0000","last_became_peered":"2023-03-03T06:18:05.375857+0000","last_unstale":"2023-03-03T06:18:07.087593+0000","last_undegraded":"2023-03-03T06:18:07.087593+0000","last_fullsized":"2023-03-03T06:18:07.087593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767254+0000","last_change":"2023-03-03T06:18:05.775649+0000","last_active":"2023-03-03T06:18:06.767254+0000","last_peered":"2023-03-03T06:18:06.767254+0000","last_clean":"2023-03-03T06:18:06.767254+0000","last_became_active":"2023-03-03T06:18:05.775370+0000","last_became_peered":"2023-03-03T06:18:05.775370+0000","last_unstale":"2023-03-03T06:18:06.767254+0000","last_undegraded":"2023-03-03T06:18:06.767254+0000","last_fullsized":"2023-03-03T06:18:06.767254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766285+0000","last_change":"2023-03-03T06:18:05.377478+0000","last_active":"2023-03-03T06:18:06.766285+0000","last_peered":"2023-03-03T06:18:06.766285+0000","last_clean":"2023-03-03T06:18:06.766285+0000","last_became_active":"2023-03-03T06:18:05.377177+0000","last_became_peered":"2023-03-03T06:18:05.377177+0000","last_unstale":"2023-03-03T06:18:06.766285+0000","last_undegraded":"2023-03-03T06:18:06.766285+0000","last_fullsized":"2023-03-03T06:18:06.766285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:42.138865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087675+0000","last_change":"2023-03-03T06:18:05.771983+0000","last_active":"2023-03-03T06:18:07.087675+0000","last_peered":"2023-03-03T06:18:07.087675+0000","last_clean":"2023-03-03T06:18:07.087675+0000","last_became_active":"2023-03-03T06:18:05.771526+0000","last_became_peered":"2023-03-03T06:18:05.771526+0000","last_unstale":"2023-03-03T06:18:07.087675+0000","last_undegraded":"2023-03-03T06:18:07.087675+0000","last_fullsized":"2023-03-03T06:18:07.087675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089756+0000","last_change":"2023-03-03T06:18:05.379123+0000","last_active":"2023-03-03T06:18:07.089756+0000","last_peered":"2023-03-03T06:18:07.089756+0000","last_clean":"2023-03-03T06:18:07.089756+0000","last_became_active":"2023-03-03T06:18:05.378826+0000","last_became_peered":"2023-03-03T06:18:05.378826+0000","last_unstale":"2023-03-03T06:18:07.089756+0000","last_undegraded":"2023-03-03T06:18:07.089756+0000","last_fullsized":"2023-03-03T06:18:07.089756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761645+0000","last_change":"2023-03-03T06:18:05.766998+0000","last_active":"2023-03-03T06:18:06.761645+0000","last_peered":"2023-03-03T06:18:06.761645+0000","last_clean":"2023-03-03T06:18:06.761645+0000","last_became_active":"2023-03-03T06:18:05.764785+0000","last_became_peered":"2023-03-03T06:18:05.764785+0000","last_unstale":"2023-03-03T06:18:06.761645+0000","last_undegraded":"2023-03-03T06:18:06.761645+0000","last_fullsized":"2023-03-03T06:18:06.761645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767168+0000","last_change":"2023-03-03T06:18:05.377431+0000","last_active":"2023-03-03T06:18:06.767168+0000","last_peered":"2023-03-03T06:18:06.767168+0000","last_clean":"2023-03-03T06:18:06.767168+0000","last_became_active":"2023-03-03T06:18:05.377107+0000","last_became_peered":"2023-03-03T06:18:05.377107+0000","last_unstale":"2023-03-03T06:18:06.767168+0000","last_undegraded":"2023-03-03T06:18:06.767168+0000","last_fullsized":"2023-03-03T06:18:06.767168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766647+0000","last_change":"2023-03-03T06:18:05.774893+0000","last_active":"2023-03-03T06:18:06.766647+0000","last_peered":"2023-03-03T06:18:06.766647+0000","last_clean":"2023-03-03T06:18:06.766647+0000","last_became_active":"2023-03-03T06:18:05.774256+0000","last_became_peered":"2023-03-03T06:18:05.774256+0000","last_unstale":"2023-03-03T06:18:06.766647+0000","last_undegraded":"2023-03-03T06:18:06.766647+0000","last_fullsized":"2023-03-03T06:18:06.766647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766744+0000","last_change":"2023-03-03T06:18:05.377497+0000","last_active":"2023-03-03T06:18:06.766744+0000","last_peered":"2023-03-03T06:18:06.766744+0000","last_clean":"2023-03-03T06:18:06.766744+0000","last_became_active":"2023-03-03T06:18:05.377215+0000","last_became_peered":"2023-03-03T06:18:05.377215+0000","last_unstale":"2023-03-03T06:18:06.766744+0000","last_undegraded":"2023-03-03T06:18:06.766744+0000","last_fullsized":"2023-03-03T06:18:06.766744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088051+0000","last_change":"2023-03-03T06:18:05.781538+0000","last_active":"2023-03-03T06:18:07.088051+0000","last_peered":"2023-03-03T06:18:07.088051+0000","last_clean":"2023-03-03T06:18:07.088051+0000","last_became_active":"2023-03-03T06:18:05.781285+0000","last_became_peered":"2023-03-03T06:18:05.781285+0000","last_unstale":"2023-03-03T06:18:07.088051+0000","last_undegraded":"2023-03-03T06:18:07.088051+0000","last_fullsized":"2023-03-03T06:18:07.088051+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767086+0000","last_change":"2023-03-03T06:18:05.369584+0000","last_active":"2023-03-03T06:18:06.767086+0000","last_peered":"2023-03-03T06:18:06.767086+0000","last_clean":"2023-03-03T06:18:06.767086+0000","last_became_active":"2023-03-03T06:18:05.368904+0000","last_became_peered":"2023-03-03T06:18:05.368904+0000","last_unstale":"2023-03-03T06:18:06.767086+0000","last_undegraded":"2023-03-03T06:18:06.767086+0000","last_fullsized":"2023-03-03T06:18:06.767086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:23.861622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089809+0000","last_change":"2023-03-03T06:18:05.779823+0000","last_active":"2023-03-03T06:18:07.089809+0000","last_peered":"2023-03-03T06:18:07.089809+0000","last_clean":"2023-03-03T06:18:07.089809+0000","last_became_active":"2023-03-03T06:18:05.779561+0000","last_became_peered":"2023-03-03T06:18:05.779561+0000","last_unstale":"2023-03-03T06:18:07.089809+0000","last_undegraded":"2023-03-03T06:18:07.089809+0000","last_fullsized":"2023-03-03T06:18:07.089809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:54.968231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767196+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:06.767196+0000","last_peered":"2023-03-03T06:18:06.767196+0000","last_clean":"2023-03-03T06:18:06.767196+0000","last_became_active":"2023-03-03T06:18:05.372057+0000","last_became_peered":"2023-03-03T06:18:05.372057+0000","last_unstale":"2023-03-03T06:18:06.767196+0000","last_undegraded":"2023-03-03T06:18:06.767196+0000","last_fullsized":"2023-03-03T06:18:06.767196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087624+0000","last_change":"2023-03-03T06:18:05.780035+0000","last_active":"2023-03-03T06:18:07.087624+0000","last_peered":"2023-03-03T06:18:07.087624+0000","last_clean":"2023-03-03T06:18:07.087624+0000","last_became_active":"2023-03-03T06:18:05.779650+0000","last_became_peered":"2023-03-03T06:18:05.779650+0000","last_unstale":"2023-03-03T06:18:07.087624+0000","last_undegraded":"2023-03-03T06:18:07.087624+0000","last_fullsized":"2023-03-03T06:18:07.087624+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761617+0000","last_change":"2023-03-03T06:18:05.379968+0000","last_active":"2023-03-03T06:18:06.761617+0000","last_peered":"2023-03-03T06:18:06.761617+0000","last_clean":"2023-03-03T06:18:06.761617+0000","last_became_active":"2023-03-03T06:18:05.379635+0000","last_became_peered":"2023-03-03T06:18:05.379635+0000","last_unstale":"2023-03-03T06:18:06.761617+0000","last_undegraded":"2023-03-03T06:18:06.761617+0000","last_fullsized":"2023-03-03T06:18:06.761617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088190+0000","last_change":"2023-03-03T06:18:05.768972+0000","last_active":"2023-03-03T06:18:07.088190+0000","last_peered":"2023-03-03T06:18:07.088190+0000","last_clean":"2023-03-03T06:18:07.088190+0000","last_became_active":"2023-03-03T06:18:05.768647+0000","last_became_peered":"2023-03-03T06:18:05.768647+0000","last_unstale":"2023-03-03T06:18:07.088190+0000","last_undegraded":"2023-03-03T06:18:07.088190+0000","last_fullsized":"2023-03-03T06:18:07.088190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:15:35.033362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089794+0000","last_change":"2023-03-03T06:18:05.378643+0000","last_active":"2023-03-03T06:18:07.089794+0000","last_peered":"2023-03-03T06:18:07.089794+0000","last_clean":"2023-03-03T06:18:07.089794+0000","last_became_active":"2023-03-03T06:18:05.378333+0000","last_became_peered":"2023-03-03T06:18:05.378333+0000","last_unstale":"2023-03-03T06:18:07.089794+0000","last_undegraded":"2023-03-03T06:18:07.089794+0000","last_fullsized":"2023-03-03T06:18:07.089794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088249+0000","last_change":"2023-03-03T06:18:05.774576+0000","last_active":"2023-03-03T06:18:07.088249+0000","last_peered":"2023-03-03T06:18:07.088249+0000","last_clean":"2023-03-03T06:18:07.088249+0000","last_became_active":"2023-03-03T06:18:05.774278+0000","last_became_peered":"2023-03-03T06:18:05.774278+0000","last_unstale":"2023-03-03T06:18:07.088249+0000","last_undegraded":"2023-03-03T06:18:07.088249+0000","last_fullsized":"2023-03-03T06:18:07.088249+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:20.212207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088073+0000","last_change":"2023-03-03T06:18:05.374483+0000","last_active":"2023-03-03T06:18:07.088073+0000","last_peered":"2023-03-03T06:18:07.088073+0000","last_clean":"2023-03-03T06:18:07.088073+0000","last_became_active":"2023-03-03T06:18:05.374224+0000","last_became_peered":"2023-03-03T06:18:05.374224+0000","last_unstale":"2023-03-03T06:18:07.088073+0000","last_undegraded":"2023-03-03T06:18:07.088073+0000","last_fullsized":"2023-03-03T06:18:07.088073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088262+0000","last_change":"2023-03-03T06:18:05.779652+0000","last_active":"2023-03-03T06:18:07.088262+0000","last_peered":"2023-03-03T06:18:07.088262+0000","last_clean":"2023-03-03T06:18:07.088262+0000","last_became_active":"2023-03-03T06:18:05.778557+0000","last_became_peered":"2023-03-03T06:18:05.778557+0000","last_unstale":"2023-03-03T06:18:07.088262+0000","last_undegraded":"2023-03-03T06:18:07.088262+0000","last_fullsized":"2023-03-03T06:18:07.088262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761087+0000","last_change":"2023-03-03T06:18:05.373177+0000","last_active":"2023-03-03T06:18:06.761087+0000","last_peered":"2023-03-03T06:18:06.761087+0000","last_clean":"2023-03-03T06:18:06.761087+0000","last_became_active":"2023-03-03T06:18:05.372879+0000","last_became_peered":"2023-03-03T06:18:05.372879+0000","last_unstale":"2023-03-03T06:18:06.761087+0000","last_undegraded":"2023-03-03T06:18:06.761087+0000","last_fullsized":"2023-03-03T06:18:06.761087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086271+0000","last_change":"2023-03-03T06:18:05.778960+0000","last_active":"2023-03-03T06:18:07.086271+0000","last_peered":"2023-03-03T06:18:07.086271+0000","last_clean":"2023-03-03T06:18:07.086271+0000","last_became_active":"2023-03-03T06:18:05.777570+0000","last_became_peered":"2023-03-03T06:18:05.777570+0000","last_unstale":"2023-03-03T06:18:07.086271+0000","last_undegraded":"2023-03-03T06:18:07.086271+0000","last_fullsized":"2023-03-03T06:18:07.086271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.775376+0000","last_change":"2023-03-03T06:18:00.701360+0000","last_active":"2023-03-03T06:18:05.775376+0000","last_peered":"2023-03-03T06:18:05.775376+0000","last_clean":"2023-03-03T06:18:05.775376+0000","last_became_active":"2023-03-03T06:18:00.701042+0000","last_became_peered":"2023-03-03T06:18:00.701042+0000","last_unstale":"2023-03-03T06:18:05.775376+0000","last_undegraded":"2023-03-03T06:18:05.775376+0000","last_fullsized":"2023-03-03T06:18:05.775376+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:10.253889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767335+0000","last_change":"2023-03-03T06:18:05.372375+0000","last_active":"2023-03-03T06:18:06.767335+0000","last_peered":"2023-03-03T06:18:06.767335+0000","last_clean":"2023-03-03T06:18:06.767335+0000","last_became_active":"2023-03-03T06:18:05.371680+0000","last_became_peered":"2023-03-03T06:18:05.371680+0000","last_unstale":"2023-03-03T06:18:06.767335+0000","last_undegraded":"2023-03-03T06:18:06.767335+0000","last_fullsized":"2023-03-03T06:18:06.767335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086177+0000","last_change":"2023-03-03T06:18:05.778931+0000","last_active":"2023-03-03T06:18:07.086177+0000","last_peered":"2023-03-03T06:18:07.086177+0000","last_clean":"2023-03-03T06:18:07.086177+0000","last_became_active":"2023-03-03T06:18:05.778723+0000","last_became_peered":"2023-03-03T06:18:05.778723+0000","last_unstale":"2023-03-03T06:18:07.086177+0000","last_undegraded":"2023-03-03T06:18:07.086177+0000","last_fullsized":"2023-03-03T06:18:07.086177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.764427+0000","last_change":"2023-03-03T06:18:00.351410+0000","last_active":"2023-03-03T06:18:05.764427+0000","last_peered":"2023-03-03T06:18:05.764427+0000","last_clean":"2023-03-03T06:18:05.764427+0000","last_became_active":"2023-03-03T06:18:00.351036+0000","last_became_peered":"2023-03-03T06:18:00.351036+0000","last_unstale":"2023-03-03T06:18:05.764427+0000","last_undegraded":"2023-03-03T06:18:05.764427+0000","last_fullsized":"2023-03-03T06:18:05.764427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088156+0000","last_change":"2023-03-03T06:18:05.378884+0000","last_active":"2023-03-03T06:18:07.088156+0000","last_peered":"2023-03-03T06:18:07.088156+0000","last_clean":"2023-03-03T06:18:07.088156+0000","last_became_active":"2023-03-03T06:18:05.378597+0000","last_became_peered":"2023-03-03T06:18:05.378597+0000","last_unstale":"2023-03-03T06:18:07.088156+0000","last_undegraded":"2023-03-03T06:18:07.088156+0000","last_fullsized":"2023-03-03T06:18:07.088156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086204+0000","last_change":"2023-03-03T06:18:05.777398+0000","last_active":"2023-03-03T06:18:07.086204+0000","last_peered":"2023-03-03T06:18:07.086204+0000","last_clean":"2023-03-03T06:18:07.086204+0000","last_became_active":"2023-03-03T06:18:05.777065+0000","last_became_peered":"2023-03-03T06:18:05.777065+0000","last_unstale":"2023-03-03T06:18:07.086204+0000","last_undegraded":"2023-03-03T06:18:07.086204+0000","last_fullsized":"2023-03-03T06:18:07.086204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.760094+0000","last_change":"2023-03-03T06:18:00.351233+0000","last_active":"2023-03-03T06:18:05.760094+0000","last_peered":"2023-03-03T06:18:05.760094+0000","last_clean":"2023-03-03T06:18:05.760094+0000","last_became_active":"2023-03-03T06:18:00.350944+0000","last_became_peered":"2023-03-03T06:18:00.350944+0000","last_unstale":"2023-03-03T06:18:05.760094+0000","last_undegraded":"2023-03-03T06:18:05.760094+0000","last_fullsized":"2023-03-03T06:18:05.760094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:08:23.569811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086137+0000","last_change":"2023-03-03T06:18:05.374876+0000","last_active":"2023-03-03T06:18:07.086137+0000","last_peered":"2023-03-03T06:18:07.086137+0000","last_clean":"2023-03-03T06:18:07.086137+0000","last_became_active":"2023-03-03T06:18:05.374266+0000","last_became_peered":"2023-03-03T06:18:05.374266+0000","last_unstale":"2023-03-03T06:18:07.086137+0000","last_undegraded":"2023-03-03T06:18:07.086137+0000","last_fullsized":"2023-03-03T06:18:07.086137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086261+0000","last_change":"2023-03-03T06:18:05.779139+0000","last_active":"2023-03-03T06:18:07.086261+0000","last_peered":"2023-03-03T06:18:07.086261+0000","last_clean":"2023-03-03T06:18:07.086261+0000","last_became_active":"2023-03-03T06:18:05.777315+0000","last_became_peered":"2023-03-03T06:18:05.777315+0000","last_unstale":"2023-03-03T06:18:07.086261+0000","last_undegraded":"2023-03-03T06:18:07.086261+0000","last_fullsized":"2023-03-03T06:18:07.086261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.762243+0000","last_change":"2023-03-03T06:18:00.351257+0000","last_active":"2023-03-03T06:18:05.762243+0000","last_peered":"2023-03-03T06:18:05.762243+0000","last_clean":"2023-03-03T06:18:05.762243+0000","last_became_active":"2023-03-03T06:18:00.350933+0000","last_became_peered":"2023-03-03T06:18:00.350933+0000","last_unstale":"2023-03-03T06:18:05.762243+0000","last_undegraded":"2023-03-03T06:18:05.762243+0000","last_fullsized":"2023-03-03T06:18:05.762243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758385+0000","last_change":"2023-03-03T06:17:54.307666+0000","last_active":"2023-03-03T06:18:05.758385+0000","last_peered":"2023-03-03T06:18:05.758385+0000","last_clean":"2023-03-03T06:18:05.758385+0000","last_became_active":"2023-03-03T06:17:54.306552+0000","last_became_peered":"2023-03-03T06:17:54.306552+0000","last_unstale":"2023-03-03T06:18:05.758385+0000","last_undegraded":"2023-03-03T06:18:05.758385+0000","last_fullsized":"2023-03-03T06:18:05.758385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_clean_scrub_stamp":"2023-03-03T06:17:53.091715+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:56:41.267904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761267+0000","last_change":"2023-03-03T06:18:05.366688+0000","last_active":"2023-03-03T06:18:06.761267+0000","last_peered":"2023-03-03T06:18:06.761267+0000","last_clean":"2023-03-03T06:18:06.761267+0000","last_became_active":"2023-03-03T06:18:05.366412+0000","last_became_peered":"2023-03-03T06:18:05.366412+0000","last_unstale":"2023-03-03T06:18:06.761267+0000","last_undegraded":"2023-03-03T06:18:06.761267+0000","last_fullsized":"2023-03-03T06:18:06.761267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089833+0000","last_change":"2023-03-03T06:18:05.781486+0000","last_active":"2023-03-03T06:18:07.089833+0000","last_peered":"2023-03-03T06:18:07.089833+0000","last_clean":"2023-03-03T06:18:07.089833+0000","last_became_active":"2023-03-03T06:18:05.781220+0000","last_became_peered":"2023-03-03T06:18:05.781220+0000","last_unstale":"2023-03-03T06:18:07.089833+0000","last_undegraded":"2023-03-03T06:18:07.089833+0000","last_fullsized":"2023-03-03T06:18:07.089833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:39.387687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758932+0000","last_change":"2023-03-03T06:18:00.345967+0000","last_active":"2023-03-03T06:18:05.758932+0000","last_peered":"2023-03-03T06:18:05.758932+0000","last_clean":"2023-03-03T06:18:05.758932+0000","last_became_active":"2023-03-03T06:18:00.345666+0000","last_became_peered":"2023-03-03T06:18:00.345666+0000","last_unstale":"2023-03-03T06:18:05.758932+0000","last_undegraded":"2023-03-03T06:18:05.758932+0000","last_fullsized":"2023-03-03T06:18:05.758932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087574+0000","last_change":"2023-03-03T06:18:05.379214+0000","last_active":"2023-03-03T06:18:07.087574+0000","last_peered":"2023-03-03T06:18:07.087574+0000","last_clean":"2023-03-03T06:18:07.087574+0000","last_became_active":"2023-03-03T06:18:05.378867+0000","last_became_peered":"2023-03-03T06:18:05.378867+0000","last_unstale":"2023-03-03T06:18:07.087574+0000","last_undegraded":"2023-03-03T06:18:07.087574+0000","last_fullsized":"2023-03-03T06:18:07.087574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087711+0000","last_change":"2023-03-03T06:18:05.782275+0000","last_active":"2023-03-03T06:18:07.087711+0000","last_peered":"2023-03-03T06:18:07.087711+0000","last_clean":"2023-03-03T06:18:07.087711+0000","last_became_active":"2023-03-03T06:18:05.781968+0000","last_became_peered":"2023-03-03T06:18:05.781968+0000","last_unstale":"2023-03-03T06:18:07.087711+0000","last_undegraded":"2023-03-03T06:18:07.087711+0000","last_fullsized":"2023-03-03T06:18:07.087711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.759159+0000","last_change":"2023-03-03T06:18:00.701652+0000","last_active":"2023-03-03T06:18:05.759159+0000","last_peered":"2023-03-03T06:18:05.759159+0000","last_clean":"2023-03-03T06:18:05.759159+0000","last_became_active":"2023-03-03T06:18:00.701382+0000","last_became_peered":"2023-03-03T06:18:00.701382+0000","last_unstale":"2023-03-03T06:18:05.759159+0000","last_undegraded":"2023-03-03T06:18:05.759159+0000","last_fullsized":"2023-03-03T06:18:05.759159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:50:16.060640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086214+0000","last_change":"2023-03-03T06:18:05.374785+0000","last_active":"2023-03-03T06:18:07.086214+0000","last_peered":"2023-03-03T06:18:07.086214+0000","last_clean":"2023-03-03T06:18:07.086214+0000","last_became_active":"2023-03-03T06:18:05.374322+0000","last_became_peered":"2023-03-03T06:18:05.374322+0000","last_unstale":"2023-03-03T06:18:07.086214+0000","last_undegraded":"2023-03-03T06:18:07.086214+0000","last_fullsized":"2023-03-03T06:18:07.086214+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087664+0000","last_change":"2023-03-03T06:18:05.775603+0000","last_active":"2023-03-03T06:18:07.087664+0000","last_peered":"2023-03-03T06:18:07.087664+0000","last_clean":"2023-03-03T06:18:07.087664+0000","last_became_active":"2023-03-03T06:18:05.775350+0000","last_became_peered":"2023-03-03T06:18:05.775350+0000","last_unstale":"2023-03-03T06:18:07.087664+0000","last_undegraded":"2023-03-03T06:18:07.087664+0000","last_fullsized":"2023-03-03T06:18:07.087664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761609+0000","last_change":"2023-03-03T06:18:05.371388+0000","last_active":"2023-03-03T06:18:06.761609+0000","last_peered":"2023-03-03T06:18:06.761609+0000","last_clean":"2023-03-03T06:18:06.761609+0000","last_became_active":"2023-03-03T06:18:05.371026+0000","last_became_peered":"2023-03-03T06:18:05.371026+0000","last_unstale":"2023-03-03T06:18:06.761609+0000","last_undegraded":"2023-03-03T06:18:06.761609+0000","last_fullsized":"2023-03-03T06:18:06.761609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.769697+0000","last_change":"2023-03-03T06:18:00.349337+0000","last_active":"2023-03-03T06:18:05.769697+0000","last_peered":"2023-03-03T06:18:05.769697+0000","last_clean":"2023-03-03T06:18:05.769697+0000","last_became_active":"2023-03-03T06:18:00.349101+0000","last_became_peered":"2023-03-03T06:18:00.349101+0000","last_unstale":"2023-03-03T06:18:05.769697+0000","last_undegraded":"2023-03-03T06:18:05.769697+0000","last_fullsized":"2023-03-03T06:18:05.769697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761022+0000","last_change":"2023-03-03T06:18:05.762420+0000","last_active":"2023-03-03T06:18:06.761022+0000","last_peered":"2023-03-03T06:18:06.761022+0000","last_clean":"2023-03-03T06:18:06.761022+0000","last_became_active":"2023-03-03T06:18:05.760911+0000","last_became_peered":"2023-03-03T06:18:05.760911+0000","last_unstale":"2023-03-03T06:18:06.761022+0000","last_undegraded":"2023-03-03T06:18:06.761022+0000","last_fullsized":"2023-03-03T06:18:06.761022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.766068+0000","last_change":"2023-03-03T06:18:00.352230+0000","last_active":"2023-03-03T06:18:05.766068+0000","last_peered":"2023-03-03T06:18:05.766068+0000","last_clean":"2023-03-03T06:18:05.766068+0000","last_became_active":"2023-03-03T06:18:00.351881+0000","last_became_peered":"2023-03-03T06:18:00.351881+0000","last_unstale":"2023-03-03T06:18:05.766068+0000","last_undegraded":"2023-03-03T06:18:05.766068+0000","last_fullsized":"2023-03-03T06:18:05.766068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:18:24.984007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.567915+0000","last_change":"2023-03-03T06:18:05.382451+0000","last_active":"2023-03-03T06:18:07.567915+0000","last_peered":"2023-03-03T06:18:07.567915+0000","last_clean":"2023-03-03T06:18:07.567915+0000","last_became_active":"2023-03-03T06:18:05.382112+0000","last_became_peered":"2023-03-03T06:18:05.382112+0000","last_unstale":"2023-03-03T06:18:07.567915+0000","last_undegraded":"2023-03-03T06:18:07.567915+0000","last_fullsized":"2023-03-03T06:18:07.567915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088062+0000","last_change":"2023-03-03T06:18:05.777771+0000","last_active":"2023-03-03T06:18:07.088062+0000","last_peered":"2023-03-03T06:18:07.088062+0000","last_clean":"2023-03-03T06:18:07.088062+0000","last_became_active":"2023-03-03T06:18:05.777457+0000","last_became_peered":"2023-03-03T06:18:05.777457+0000","last_unstale":"2023-03-03T06:18:07.088062+0000","last_undegraded":"2023-03-03T06:18:07.088062+0000","last_fullsized":"2023-03-03T06:18:07.088062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:24.867158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087635+0000","last_change":"2023-03-03T06:18:05.382340+0000","last_active":"2023-03-03T06:18:07.087635+0000","last_peered":"2023-03-03T06:18:07.087635+0000","last_clean":"2023-03-03T06:18:07.087635+0000","last_became_active":"2023-03-03T06:18:05.382051+0000","last_became_peered":"2023-03-03T06:18:05.382051+0000","last_unstale":"2023-03-03T06:18:07.087635+0000","last_undegraded":"2023-03-03T06:18:07.087635+0000","last_fullsized":"2023-03-03T06:18:07.087635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089777+0000","last_change":"2023-03-03T06:18:05.778551+0000","last_active":"2023-03-03T06:18:07.089777+0000","last_peered":"2023-03-03T06:18:07.089777+0000","last_clean":"2023-03-03T06:18:07.089777+0000","last_became_active":"2023-03-03T06:18:05.778166+0000","last_became_peered":"2023-03-03T06:18:05.778166+0000","last_unstale":"2023-03-03T06:18:07.089777+0000","last_undegraded":"2023-03-03T06:18:07.089777+0000","last_fullsized":"2023-03-03T06:18:07.089777+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:39.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766897+0000","last_change":"2023-03-03T06:18:05.371014+0000","last_active":"2023-03-03T06:18:06.766897+0000","last_peered":"2023-03-03T06:18:06.766897+0000","last_clean":"2023-03-03T06:18:06.766897+0000","last_became_active":"2023-03-03T06:18:05.370730+0000","last_became_peered":"2023-03-03T06:18:05.370730+0000","last_unstale":"2023-03-03T06:18:06.766897+0000","last_undegraded":"2023-03-03T06:18:06.766897+0000","last_fullsized":"2023-03-03T06:18:06.766897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088144+0000","last_change":"2023-03-03T06:18:05.772599+0000","last_active":"2023-03-03T06:18:07.088144+0000","last_peered":"2023-03-03T06:18:07.088144+0000","last_clean":"2023-03-03T06:18:07.088144+0000","last_became_active":"2023-03-03T06:18:05.772305+0000","last_became_peered":"2023-03-03T06:18:05.772305+0000","last_unstale":"2023-03-03T06:18:07.088144+0000","last_undegraded":"2023-03-03T06:18:07.088144+0000","last_fullsized":"2023-03-03T06:18:07.088144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:14.254976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767411+0000","last_change":"2023-03-03T06:18:05.374466+0000","last_active":"2023-03-03T06:18:06.767411+0000","last_peered":"2023-03-03T06:18:06.767411+0000","last_clean":"2023-03-03T06:18:06.767411+0000","last_became_active":"2023-03-03T06:18:05.374194+0000","last_became_peered":"2023-03-03T06:18:05.374194+0000","last_unstale":"2023-03-03T06:18:06.767411+0000","last_undegraded":"2023-03-03T06:18:06.767411+0000","last_fullsized":"2023-03-03T06:18:06.767411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088131+0000","last_change":"2023-03-03T06:18:05.773340+0000","last_active":"2023-03-03T06:18:07.088131+0000","last_peered":"2023-03-03T06:18:07.088131+0000","last_clean":"2023-03-03T06:18:07.088131+0000","last_became_active":"2023-03-03T06:18:05.772391+0000","last_became_peered":"2023-03-03T06:18:05.772391+0000","last_unstale":"2023-03-03T06:18:07.088131+0000","last_undegraded":"2023-03-03T06:18:07.088131+0000","last_fullsized":"2023-03-03T06:18:07.088131+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.762090+0000","last_change":"2023-03-03T06:18:05.369911+0000","last_active":"2023-03-03T06:18:06.762090+0000","last_peered":"2023-03-03T06:18:06.762090+0000","last_clean":"2023-03-03T06:18:06.762090+0000","last_became_active":"2023-03-03T06:18:05.369605+0000","last_became_peered":"2023-03-03T06:18:05.369605+0000","last_unstale":"2023-03-03T06:18:06.762090+0000","last_undegraded":"2023-03-03T06:18:06.762090+0000","last_fullsized":"2023-03-03T06:18:06.762090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761116+0000","last_change":"2023-03-03T06:18:05.766192+0000","last_active":"2023-03-03T06:18:06.761116+0000","last_peered":"2023-03-03T06:18:06.761116+0000","last_clean":"2023-03-03T06:18:06.761116+0000","last_became_active":"2023-03-03T06:18:05.764744+0000","last_became_peered":"2023-03-03T06:18:05.764744+0000","last_unstale":"2023-03-03T06:18:06.761116+0000","last_undegraded":"2023-03-03T06:18:06.761116+0000","last_fullsized":"2023-03-03T06:18:06.761116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087752+0000","last_change":"2023-03-03T06:18:05.378275+0000","last_active":"2023-03-03T06:18:07.087752+0000","last_peered":"2023-03-03T06:18:07.087752+0000","last_clean":"2023-03-03T06:18:07.087752+0000","last_became_active":"2023-03-03T06:18:05.377803+0000","last_became_peered":"2023-03-03T06:18:05.377803+0000","last_unstale":"2023-03-03T06:18:07.087752+0000","last_undegraded":"2023-03-03T06:18:07.087752+0000","last_fullsized":"2023-03-03T06:18:07.087752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088033+0000","last_change":"2023-03-03T06:18:05.770801+0000","last_active":"2023-03-03T06:18:07.088033+0000","last_peered":"2023-03-03T06:18:07.088033+0000","last_clean":"2023-03-03T06:18:07.088033+0000","last_became_active":"2023-03-03T06:18:05.770500+0000","last_became_peered":"2023-03-03T06:18:05.770500+0000","last_unstale":"2023-03-03T06:18:07.088033+0000","last_undegraded":"2023-03-03T06:18:07.088033+0000","last_fullsized":"2023-03-03T06:18:07.088033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089779+0000","last_change":"2023-03-03T06:18:05.776954+0000","last_active":"2023-03-03T06:18:07.089779+0000","last_peered":"2023-03-03T06:18:07.089779+0000","last_clean":"2023-03-03T06:18:07.089779+0000","last_became_active":"2023-03-03T06:18:05.776665+0000","last_became_peered":"2023-03-03T06:18:05.776665+0000","last_unstale":"2023-03-03T06:18:07.089779+0000","last_undegraded":"2023-03-03T06:18:07.089779+0000","last_fullsized":"2023-03-03T06:18:07.089779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:46.917574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761055+0000","last_change":"2023-03-03T06:18:05.780029+0000","last_active":"2023-03-03T06:18:06.761055+0000","last_peered":"2023-03-03T06:18:06.761055+0000","last_clean":"2023-03-03T06:18:06.761055+0000","last_became_active":"2023-03-03T06:18:05.779805+0000","last_became_peered":"2023-03-03T06:18:05.779805+0000","last_unstale":"2023-03-03T06:18:06.761055+0000","last_undegraded":"2023-03-03T06:18:06.761055+0000","last_fullsized":"2023-03-03T06:18:06.761055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088099+0000","last_change":"2023-03-03T06:18:05.781886+0000","last_active":"2023-03-03T06:18:07.088099+0000","last_peered":"2023-03-03T06:18:07.088099+0000","last_clean":"2023-03-03T06:18:07.088099+0000","last_became_active":"2023-03-03T06:18:05.780587+0000","last_became_peered":"2023-03-03T06:18:05.780587+0000","last_unstale":"2023-03-03T06:18:07.088099+0000","last_undegraded":"2023-03-03T06:18:07.088099+0000","last_fullsized":"2023-03-03T06:18:07.088099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:02.129741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761713+0000","last_change":"2023-03-03T06:18:05.760339+0000","last_active":"2023-03-03T06:18:06.761713+0000","last_peered":"2023-03-03T06:18:06.761713+0000","last_clean":"2023-03-03T06:18:06.761713+0000","last_became_active":"2023-03-03T06:18:05.759652+0000","last_became_peered":"2023-03-03T06:18:05.759652+0000","last_unstale":"2023-03-03T06:18:06.761713+0000","last_undegraded":"2023-03-03T06:18:06.761713+0000","last_fullsized":"2023-03-03T06:18:06.761713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086186+0000","last_change":"2023-03-03T06:18:05.777608+0000","last_active":"2023-03-03T06:18:07.086186+0000","last_peered":"2023-03-03T06:18:07.086186+0000","last_clean":"2023-03-03T06:18:07.086186+0000","last_became_active":"2023-03-03T06:18:05.777310+0000","last_became_peered":"2023-03-03T06:18:05.777310+0000","last_unstale":"2023-03-03T06:18:07.086186+0000","last_undegraded":"2023-03-03T06:18:07.086186+0000","last_fullsized":"2023-03-03T06:18:07.086186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9108,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362732,"statfs":{"total":96636764160,"available":96627437568,"internally_reserved":0,"allocated":864256,"data_stored":681731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7988,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363852,"statfs":{"total":96636764160,"available":96628584448,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9559999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9949999999999992}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8052,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363788,"statfs":{"total":96636764160,"available":96628518912,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0340000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1659999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.202}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.984999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.061}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9084,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362756,"statfs":{"total":96636764160,"available":96627462144,"internally_reserved":0,"allocated":839680,"data_stored":680533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7924,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363916,"statfs":{"total":96636764160,"available":96628649984,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.834}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.032}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.786}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8056,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363784,"statfs":{"total":96636764160,"available":96628514816,"internally_reserved":0,"allocated":245760,"data_stored":89689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8072,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363768,"statfs":{"total":96636764160,"available":96628498432,"internally_reserved":0,"allocated":262144,"data_stored":91203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":90235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0339999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T06:18:14.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T06:18:14.387 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.385+0000 7f397d530700 1 Processor -- start 2023-03-03T06:18:14.387 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.385+0000 7f397d530700 1 -- start start 2023-03-03T06:18:14.393 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f397d530700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 0x7f39780c2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.393 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f397d530700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3978146310 con 0x7f397814fe60 2023-03-03T06:18:14.393 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f397d530700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397806d420 con 0x7f39780c2860 2023-03-03T06:18:14.393 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f397d530700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397814cc50 con 0x7f39780bd220 2023-03-03T06:18:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f39767fc700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f397814fe60 0x7f3978154760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53602/0 (socket says 172.21.15.90:53602) 2023-03-03T06:18:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f39767fc700 1 -- 172.21.15.90:0/2524914334 learned_addr learned my addr 172.21.15.90:0/2524914334 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f39777fe700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 0x7f39780c2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971238967 0 0) 0x7f3968001110 con 0x7f397814fe60 2023-03-03T06:18:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39600035b0 con 0x7f397814fe60 2023-03-03T06:18:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 991139020 0 0) 0x7f39600035b0 con 0x7f397814fe60 2023-03-03T06:18:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 msgr2=0x7f39780c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 0x7f39780c2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 >> v1:172.21.15.121:6789/0 conn(0x7f39780c2860 legacy=0x7f397814f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39780bd770 con 0x7f397814fe60 2023-03-03T06:18:14.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f3968001800 con 0x7f397814fe60 2023-03-03T06:18:14.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.386+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 >> v1:172.21.15.90:6789/0 conn(0x7f397814fe60 legacy=0x7f3978154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.396 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f3960003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.397 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3960004850 0x7f3960004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.397 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 0x7f3960005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.397 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397814cc50 con 0x7f3960003aa0 2023-03-03T06:18:14.397 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396000baf0 con 0x7f3960004850 2023-03-03T06:18:14.397 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396000bcb0 con 0x7f39780bd220 2023-03-03T06:18:14.397 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3976ffd700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f3960003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f39767fc700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3960004850 0x7f3960004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f39777fe700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 0x7f3960005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3976ffd700 1 -- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 msgr2=0x7f3960005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3976ffd700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39780bd220 0x7f3960005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3976ffd700 1 -- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3960004850 msgr2=0x7f3960004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3976ffd700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3960004850 0x7f3960004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3976ffd700 1 -- 172.21.15.90:0/2524914334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39600035b0 con 0x7f3960003aa0 2023-03-03T06:18:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3976ffd700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f3960003e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f396c0018b0 tx=0x7f3960005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20601605070b90f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c001370 con 0x7f3960003aa0 2023-03-03T06:18:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c00c6b0 con 0x7f3960003aa0 2023-03-03T06:18:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.387+0000 7f3975ffb700 1 -- 172.21.15.90:0/2524914334 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c00ca60 con 0x7f3960003aa0 2023-03-03T06:18:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.388+0000 7f397d530700 1 -- 172.21.15.90:0/2524914334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 msgr2=0x7f3960003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.400 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.388+0000 7f397d530700 1 --2- 172.21.15.90:0/2524914334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f3960003e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f396c0018b0 tx=0x7f3960005880 comp rx=0 tx=0).stop 2023-03-03T06:18:14.400 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.388+0000 7f39777fe700 1 -- 172.21.15.90:0/2524914334 reap_dead start 2023-03-03T06:18:14.400 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.388+0000 7f397d530700 1 -- 172.21.15.90:0/2524914334 shutdown_connections 2023-03-03T06:18:14.400 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.388+0000 7f397d530700 1 -- 172.21.15.90:0/2524914334 >> 172.21.15.90:0/2524914334 conn(0x7f397806c0d0 msgr2=0x7f39780c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:14.400 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.388+0000 7f397d530700 1 -- 172.21.15.90:0/2524914334 shutdown_connections 2023-03-03T06:18:14.401 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.388+0000 7f397d530700 1 -- 172.21.15.90:0/2524914334 wait complete. 2023-03-03T06:18:14.401 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.389+0000 7f397d530700 1 Processor -- start 2023-03-03T06:18:14.401 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.389+0000 7f397d530700 1 -- start start 2023-03-03T06:18:14.401 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f39781e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3978146eb0 0x7f3978147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3978158780 0x7f3978158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3978147860 con 0x7f3960003aa0 2023-03-03T06:18:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39780bc150 con 0x7f3978158780 2023-03-03T06:18:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3978149670 con 0x7f3978146eb0 2023-03-03T06:18:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f39781e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f39781e6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34406/0 (socket says 172.21.15.90:34406) 2023-03-03T06:18:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 -- 172.21.15.90:0/3414739034 learned_addr learned my addr 172.21.15.90:0/3414739034 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f39767fc700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3978146eb0 0x7f3978147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f39777fe700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3978158780 0x7f3978158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 -- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3978146eb0 msgr2=0x7f3978147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3978146eb0 0x7f3978147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 -- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3978158780 msgr2=0x7f3978158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3978158780 0x7f3978158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 -- 172.21.15.90:0/3414739034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39780bdd30 con 0x7f3960003aa0 2023-03-03T06:18:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f3976ffd700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f39781e6170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f396c000c00 tx=0x7f396c001110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f395f7fe700 1 -- 172.21.15.90:0/3414739034 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c001370 con 0x7f3960003aa0 2023-03-03T06:18:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f395f7fe700 1 -- 172.21.15.90:0/3414739034 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c005570 con 0x7f3960003aa0 2023-03-03T06:18:14.405 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39780bda50 con 0x7f3960003aa0 2023-03-03T06:18:14.405 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f395f7fe700 1 -- 172.21.15.90:0/3414739034 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c00ce70 con 0x7f3960003aa0 2023-03-03T06:18:14.405 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.390+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39780723b0 con 0x7f3960003aa0 2023-03-03T06:18:14.405 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.391+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397814d510 con 0x7f3960003aa0 2023-03-03T06:18:14.406 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.392+0000 7f395f7fe700 1 -- 172.21.15.90:0/3414739034 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f396c010050 con 0x7f3960003aa0 2023-03-03T06:18:14.407 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.392+0000 7f395f7fe700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f39640682f0 0x7f396406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.407 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.393+0000 7f39767fc700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f39640682f0 0x7f396406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.407 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.393+0000 7f395f7fe700 1 -- 172.21.15.90:0/3414739034 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f396c087060 con 0x7f3960003aa0 2023-03-03T06:18:14.407 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.393+0000 7f39767fc700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f39640682f0 0x7f396406a7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3968007fd0 tx=0x7f3968007e00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.408 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.396+0000 7f395f7fe700 1 -- 172.21.15.90:0/3414739034 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f396c053900 con 0x7f3960003aa0 2023-03-03T06:18:14.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.576+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 --> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f397804e750 con 0x7f39640682f0 2023-03-03T06:18:14.585 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.584+0000 7f395f7fe700 1 -- 172.21.15.90:0/3414739034 <== mgr.4108 v2:172.21.15.90:6832/96475 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286476 (secure 0 0 0) 0x7f397804e750 con 0x7f39640682f0 2023-03-03T06:18:14.589 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:14.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f39640682f0 msgr2=0x7f396406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f39640682f0 0x7f396406a7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3968007fd0 tx=0x7f3968007e00 comp rx=0 tx=0).stop 2023-03-03T06:18:14.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 msgr2=0x7f39781e6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f39781e6170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f396c000c00 tx=0x7f396c001110 comp rx=0 tx=0).stop 2023-03-03T06:18:14.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 shutdown_connections 2023-03-03T06:18:14.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3978158780 0x7f3978158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f39640682f0 0x7f396406a7a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3960003aa0 0x7f39781e6170 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 --2- 172.21.15.90:0/3414739034 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3978146eb0 0x7f3978147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 >> 172.21.15.90:0/3414739034 conn(0x7f397806c0d0 msgr2=0x7f397806a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:14.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 shutdown_connections 2023-03-03T06:18:14.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.587+0000 7f397d530700 1 -- 172.21.15.90:0/3414739034 wait complete. 2023-03-03T06:18:14.593 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-03-03T06:18:14.601 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-03-03T06:18:14.078244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66280,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":754908440,"statfs":{"total":773094113280,"available":773026242560,"internally_reserved":0,"allocated":3186688,"data_stored":1902188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.323681"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089798+0000","last_change":"2023-03-03T06:18:05.768223+0000","last_active":"2023-03-03T06:18:07.089798+0000","last_peered":"2023-03-03T06:18:07.089798+0000","last_clean":"2023-03-03T06:18:07.089798+0000","last_became_active":"2023-03-03T06:18:05.767943+0000","last_became_peered":"2023-03-03T06:18:05.767943+0000","last_unstale":"2023-03-03T06:18:07.089798+0000","last_undegraded":"2023-03-03T06:18:07.089798+0000","last_fullsized":"2023-03-03T06:18:07.089798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:38.546721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767372+0000","last_change":"2023-03-03T06:18:05.779938+0000","last_active":"2023-03-03T06:18:06.767372+0000","last_peered":"2023-03-03T06:18:06.767372+0000","last_clean":"2023-03-03T06:18:06.767372+0000","last_became_active":"2023-03-03T06:18:05.779669+0000","last_became_peered":"2023-03-03T06:18:05.779669+0000","last_unstale":"2023-03-03T06:18:06.767372+0000","last_undegraded":"2023-03-03T06:18:06.767372+0000","last_fullsized":"2023-03-03T06:18:06.767372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088222+0000","last_change":"2023-03-03T06:18:05.773328+0000","last_active":"2023-03-03T06:18:07.088222+0000","last_peered":"2023-03-03T06:18:07.088222+0000","last_clean":"2023-03-03T06:18:07.088222+0000","last_became_active":"2023-03-03T06:18:05.773065+0000","last_became_peered":"2023-03-03T06:18:05.773065+0000","last_unstale":"2023-03-03T06:18:07.088222+0000","last_undegraded":"2023-03-03T06:18:07.088222+0000","last_fullsized":"2023-03-03T06:18:07.088222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:40:58.062737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089819+0000","last_change":"2023-03-03T06:18:05.773425+0000","last_active":"2023-03-03T06:18:07.089819+0000","last_peered":"2023-03-03T06:18:07.089819+0000","last_clean":"2023-03-03T06:18:07.089819+0000","last_became_active":"2023-03-03T06:18:05.772997+0000","last_became_peered":"2023-03-03T06:18:05.772997+0000","last_unstale":"2023-03-03T06:18:07.089819+0000","last_undegraded":"2023-03-03T06:18:07.089819+0000","last_fullsized":"2023-03-03T06:18:07.089819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:43.144401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087652+0000","last_change":"2023-03-03T06:18:05.771575+0000","last_active":"2023-03-03T06:18:07.087652+0000","last_peered":"2023-03-03T06:18:07.087652+0000","last_clean":"2023-03-03T06:18:07.087652+0000","last_became_active":"2023-03-03T06:18:05.771326+0000","last_became_peered":"2023-03-03T06:18:05.771326+0000","last_unstale":"2023-03-03T06:18:07.087652+0000","last_undegraded":"2023-03-03T06:18:07.087652+0000","last_fullsized":"2023-03-03T06:18:07.087652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088086+0000","last_change":"2023-03-03T06:18:05.773873+0000","last_active":"2023-03-03T06:18:07.088086+0000","last_peered":"2023-03-03T06:18:07.088086+0000","last_clean":"2023-03-03T06:18:07.088086+0000","last_became_active":"2023-03-03T06:18:05.773610+0000","last_became_peered":"2023-03-03T06:18:05.773610+0000","last_unstale":"2023-03-03T06:18:07.088086+0000","last_undegraded":"2023-03-03T06:18:07.088086+0000","last_fullsized":"2023-03-03T06:18:07.088086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T06:24:21.809616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766274+0000","last_change":"2023-03-03T06:18:05.779587+0000","last_active":"2023-03-03T06:18:06.766274+0000","last_peered":"2023-03-03T06:18:06.766274+0000","last_clean":"2023-03-03T06:18:06.766274+0000","last_became_active":"2023-03-03T06:18:05.779147+0000","last_became_peered":"2023-03-03T06:18:05.779147+0000","last_unstale":"2023-03-03T06:18:06.766274+0000","last_undegraded":"2023-03-03T06:18:06.766274+0000","last_fullsized":"2023-03-03T06:18:06.766274+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766529+0000","last_change":"2023-03-03T06:18:05.774633+0000","last_active":"2023-03-03T06:18:06.766529+0000","last_peered":"2023-03-03T06:18:06.766529+0000","last_clean":"2023-03-03T06:18:06.766529+0000","last_became_active":"2023-03-03T06:18:05.773725+0000","last_became_peered":"2023-03-03T06:18:05.773725+0000","last_unstale":"2023-03-03T06:18:06.766529+0000","last_undegraded":"2023-03-03T06:18:06.766529+0000","last_fullsized":"2023-03-03T06:18:06.766529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087720+0000","last_change":"2023-03-03T06:18:05.770503+0000","last_active":"2023-03-03T06:18:07.087720+0000","last_peered":"2023-03-03T06:18:07.087720+0000","last_clean":"2023-03-03T06:18:07.087720+0000","last_became_active":"2023-03-03T06:18:05.770093+0000","last_became_peered":"2023-03-03T06:18:05.770093+0000","last_unstale":"2023-03-03T06:18:07.087720+0000","last_undegraded":"2023-03-03T06:18:07.087720+0000","last_fullsized":"2023-03-03T06:18:07.087720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767476+0000","last_change":"2023-03-03T06:18:05.774425+0000","last_active":"2023-03-03T06:18:06.767476+0000","last_peered":"2023-03-03T06:18:06.767476+0000","last_clean":"2023-03-03T06:18:06.767476+0000","last_became_active":"2023-03-03T06:18:05.774137+0000","last_became_peered":"2023-03-03T06:18:05.774137+0000","last_unstale":"2023-03-03T06:18:06.767476+0000","last_undegraded":"2023-03-03T06:18:06.767476+0000","last_fullsized":"2023-03-03T06:18:06.767476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088236+0000","last_change":"2023-03-03T06:18:05.779203+0000","last_active":"2023-03-03T06:18:07.088236+0000","last_peered":"2023-03-03T06:18:07.088236+0000","last_clean":"2023-03-03T06:18:07.088236+0000","last_became_active":"2023-03-03T06:18:05.778887+0000","last_became_peered":"2023-03-03T06:18:05.778887+0000","last_unstale":"2023-03-03T06:18:07.088236+0000","last_undegraded":"2023-03-03T06:18:07.088236+0000","last_fullsized":"2023-03-03T06:18:07.088236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089765+0000","last_change":"2023-03-03T06:18:05.771290+0000","last_active":"2023-03-03T06:18:07.089765+0000","last_peered":"2023-03-03T06:18:07.089765+0000","last_clean":"2023-03-03T06:18:07.089765+0000","last_became_active":"2023-03-03T06:18:05.771073+0000","last_became_peered":"2023-03-03T06:18:05.771073+0000","last_unstale":"2023-03-03T06:18:07.089765+0000","last_undegraded":"2023-03-03T06:18:07.089765+0000","last_fullsized":"2023-03-03T06:18:07.089765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:01.951025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089750+0000","last_change":"2023-03-03T06:18:05.772434+0000","last_active":"2023-03-03T06:18:07.089750+0000","last_peered":"2023-03-03T06:18:07.089750+0000","last_clean":"2023-03-03T06:18:07.089750+0000","last_became_active":"2023-03-03T06:18:05.772171+0000","last_became_peered":"2023-03-03T06:18:05.772171+0000","last_unstale":"2023-03-03T06:18:07.089750+0000","last_undegraded":"2023-03-03T06:18:07.089750+0000","last_fullsized":"2023-03-03T06:18:07.089750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:43.200309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766717+0000","last_change":"2023-03-03T06:18:05.775751+0000","last_active":"2023-03-03T06:18:06.766717+0000","last_peered":"2023-03-03T06:18:06.766717+0000","last_clean":"2023-03-03T06:18:06.766717+0000","last_became_active":"2023-03-03T06:18:05.775268+0000","last_became_peered":"2023-03-03T06:18:05.775268+0000","last_unstale":"2023-03-03T06:18:06.766717+0000","last_undegraded":"2023-03-03T06:18:06.766717+0000","last_fullsized":"2023-03-03T06:18:06.766717+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766405+0000","last_change":"2023-03-03T06:18:05.779415+0000","last_active":"2023-03-03T06:18:06.766405+0000","last_peered":"2023-03-03T06:18:06.766405+0000","last_clean":"2023-03-03T06:18:06.766405+0000","last_became_active":"2023-03-03T06:18:05.778767+0000","last_became_peered":"2023-03-03T06:18:05.778767+0000","last_unstale":"2023-03-03T06:18:06.766405+0000","last_undegraded":"2023-03-03T06:18:06.766405+0000","last_fullsized":"2023-03-03T06:18:06.766405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088011+0000","last_change":"2023-03-03T06:18:05.768282+0000","last_active":"2023-03-03T06:18:07.088011+0000","last_peered":"2023-03-03T06:18:07.088011+0000","last_clean":"2023-03-03T06:18:07.088011+0000","last_became_active":"2023-03-03T06:18:05.767972+0000","last_became_peered":"2023-03-03T06:18:05.767972+0000","last_unstale":"2023-03-03T06:18:07.088011+0000","last_undegraded":"2023-03-03T06:18:07.088011+0000","last_fullsized":"2023-03-03T06:18:07.088011+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:46.885266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087730+0000","last_change":"2023-03-03T06:18:05.770349+0000","last_active":"2023-03-03T06:18:07.087730+0000","last_peered":"2023-03-03T06:18:07.087730+0000","last_clean":"2023-03-03T06:18:07.087730+0000","last_became_active":"2023-03-03T06:18:05.769801+0000","last_became_peered":"2023-03-03T06:18:05.769801+0000","last_unstale":"2023-03-03T06:18:07.087730+0000","last_undegraded":"2023-03-03T06:18:07.087730+0000","last_fullsized":"2023-03-03T06:18:07.087730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088273+0000","last_change":"2023-03-03T06:18:05.769983+0000","last_active":"2023-03-03T06:18:07.088273+0000","last_peered":"2023-03-03T06:18:07.088273+0000","last_clean":"2023-03-03T06:18:07.088273+0000","last_became_active":"2023-03-03T06:18:05.769713+0000","last_became_peered":"2023-03-03T06:18:05.769713+0000","last_unstale":"2023-03-03T06:18:07.088273+0000","last_undegraded":"2023-03-03T06:18:07.088273+0000","last_fullsized":"2023-03-03T06:18:07.088273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:58:48.612674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089788+0000","last_change":"2023-03-03T06:18:05.772674+0000","last_active":"2023-03-03T06:18:07.089788+0000","last_peered":"2023-03-03T06:18:07.089788+0000","last_clean":"2023-03-03T06:18:07.089788+0000","last_became_active":"2023-03-03T06:18:05.772420+0000","last_became_peered":"2023-03-03T06:18:05.772420+0000","last_unstale":"2023-03-03T06:18:07.089788+0000","last_undegraded":"2023-03-03T06:18:07.089788+0000","last_fullsized":"2023-03-03T06:18:07.089788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:17.201682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088180+0000","last_change":"2023-03-03T06:18:05.779465+0000","last_active":"2023-03-03T06:18:07.088180+0000","last_peered":"2023-03-03T06:18:07.088180+0000","last_clean":"2023-03-03T06:18:07.088180+0000","last_became_active":"2023-03-03T06:18:05.779177+0000","last_became_peered":"2023-03-03T06:18:05.779177+0000","last_unstale":"2023-03-03T06:18:07.088180+0000","last_undegraded":"2023-03-03T06:18:07.088180+0000","last_fullsized":"2023-03-03T06:18:07.088180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766718+0000","last_change":"2023-03-03T06:18:05.769339+0000","last_active":"2023-03-03T06:18:06.766718+0000","last_peered":"2023-03-03T06:18:06.766718+0000","last_clean":"2023-03-03T06:18:06.766718+0000","last_became_active":"2023-03-03T06:18:05.768277+0000","last_became_peered":"2023-03-03T06:18:05.768277+0000","last_unstale":"2023-03-03T06:18:06.766718+0000","last_undegraded":"2023-03-03T06:18:06.766718+0000","last_fullsized":"2023-03-03T06:18:06.766718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087614+0000","last_change":"2023-03-03T06:18:05.780097+0000","last_active":"2023-03-03T06:18:07.087614+0000","last_peered":"2023-03-03T06:18:07.087614+0000","last_clean":"2023-03-03T06:18:07.087614+0000","last_became_active":"2023-03-03T06:18:05.779838+0000","last_became_peered":"2023-03-03T06:18:05.779838+0000","last_unstale":"2023-03-03T06:18:07.087614+0000","last_undegraded":"2023-03-03T06:18:07.087614+0000","last_fullsized":"2023-03-03T06:18:07.087614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761222+0000","last_change":"2023-03-03T06:18:05.772869+0000","last_active":"2023-03-03T06:18:06.761222+0000","last_peered":"2023-03-03T06:18:06.761222+0000","last_clean":"2023-03-03T06:18:06.761222+0000","last_became_active":"2023-03-03T06:18:05.772498+0000","last_became_peered":"2023-03-03T06:18:05.772498+0000","last_unstale":"2023-03-03T06:18:06.761222+0000","last_undegraded":"2023-03-03T06:18:06.761222+0000","last_fullsized":"2023-03-03T06:18:06.761222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766506+0000","last_change":"2023-03-03T06:18:05.769326+0000","last_active":"2023-03-03T06:18:06.766506+0000","last_peered":"2023-03-03T06:18:06.766506+0000","last_clean":"2023-03-03T06:18:06.766506+0000","last_became_active":"2023-03-03T06:18:05.768781+0000","last_became_peered":"2023-03-03T06:18:05.768781+0000","last_unstale":"2023-03-03T06:18:06.766506+0000","last_undegraded":"2023-03-03T06:18:06.766506+0000","last_fullsized":"2023-03-03T06:18:06.766506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086250+0000","last_change":"2023-03-03T06:18:05.777824+0000","last_active":"2023-03-03T06:18:07.086250+0000","last_peered":"2023-03-03T06:18:07.086250+0000","last_clean":"2023-03-03T06:18:07.086250+0000","last_became_active":"2023-03-03T06:18:05.777221+0000","last_became_peered":"2023-03-03T06:18:05.777221+0000","last_unstale":"2023-03-03T06:18:07.086250+0000","last_undegraded":"2023-03-03T06:18:07.086250+0000","last_fullsized":"2023-03-03T06:18:07.086250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:24:24.655515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086239+0000","last_change":"2023-03-03T06:18:05.777434+0000","last_active":"2023-03-03T06:18:07.086239+0000","last_peered":"2023-03-03T06:18:07.086239+0000","last_clean":"2023-03-03T06:18:07.086239+0000","last_became_active":"2023-03-03T06:18:05.776857+0000","last_became_peered":"2023-03-03T06:18:05.776857+0000","last_unstale":"2023-03-03T06:18:07.086239+0000","last_undegraded":"2023-03-03T06:18:07.086239+0000","last_fullsized":"2023-03-03T06:18:07.086239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767237+0000","last_change":"2023-03-03T06:18:05.776802+0000","last_active":"2023-03-03T06:18:06.767237+0000","last_peered":"2023-03-03T06:18:06.767237+0000","last_clean":"2023-03-03T06:18:06.767237+0000","last_became_active":"2023-03-03T06:18:05.776531+0000","last_became_peered":"2023-03-03T06:18:05.776531+0000","last_unstale":"2023-03-03T06:18:06.767237+0000","last_undegraded":"2023-03-03T06:18:06.767237+0000","last_fullsized":"2023-03-03T06:18:06.767237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088118+0000","last_change":"2023-03-03T06:18:05.374916+0000","last_active":"2023-03-03T06:18:07.088118+0000","last_peered":"2023-03-03T06:18:07.088118+0000","last_clean":"2023-03-03T06:18:07.088118+0000","last_became_active":"2023-03-03T06:18:05.374663+0000","last_became_peered":"2023-03-03T06:18:05.374663+0000","last_unstale":"2023-03-03T06:18:07.088118+0000","last_undegraded":"2023-03-03T06:18:07.088118+0000","last_fullsized":"2023-03-03T06:18:07.088118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:19:22.426709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761867+0000","last_change":"2023-03-03T06:18:05.778939+0000","last_active":"2023-03-03T06:18:06.761867+0000","last_peered":"2023-03-03T06:18:06.761867+0000","last_clean":"2023-03-03T06:18:06.761867+0000","last_became_active":"2023-03-03T06:18:05.778350+0000","last_became_peered":"2023-03-03T06:18:05.778350+0000","last_unstale":"2023-03-03T06:18:06.761867+0000","last_undegraded":"2023-03-03T06:18:06.761867+0000","last_fullsized":"2023-03-03T06:18:06.761867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086156+0000","last_change":"2023-03-03T06:18:05.369187+0000","last_active":"2023-03-03T06:18:07.086156+0000","last_peered":"2023-03-03T06:18:07.086156+0000","last_clean":"2023-03-03T06:18:07.086156+0000","last_became_active":"2023-03-03T06:18:05.368706+0000","last_became_peered":"2023-03-03T06:18:05.368706+0000","last_unstale":"2023-03-03T06:18:07.086156+0000","last_undegraded":"2023-03-03T06:18:07.086156+0000","last_fullsized":"2023-03-03T06:18:07.086156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087690+0000","last_change":"2023-03-03T06:18:05.771810+0000","last_active":"2023-03-03T06:18:07.087690+0000","last_peered":"2023-03-03T06:18:07.087690+0000","last_clean":"2023-03-03T06:18:07.087690+0000","last_became_active":"2023-03-03T06:18:05.771484+0000","last_became_peered":"2023-03-03T06:18:05.771484+0000","last_unstale":"2023-03-03T06:18:07.087690+0000","last_undegraded":"2023-03-03T06:18:07.087690+0000","last_fullsized":"2023-03-03T06:18:07.087690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761169+0000","last_change":"2023-03-03T06:18:05.378838+0000","last_active":"2023-03-03T06:18:06.761169+0000","last_peered":"2023-03-03T06:18:06.761169+0000","last_clean":"2023-03-03T06:18:06.761169+0000","last_became_active":"2023-03-03T06:18:05.378525+0000","last_became_peered":"2023-03-03T06:18:05.378525+0000","last_unstale":"2023-03-03T06:18:06.761169+0000","last_undegraded":"2023-03-03T06:18:06.761169+0000","last_fullsized":"2023-03-03T06:18:06.761169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089855+0000","last_change":"2023-03-03T06:18:05.776315+0000","last_active":"2023-03-03T06:18:07.089855+0000","last_peered":"2023-03-03T06:18:07.089855+0000","last_clean":"2023-03-03T06:18:07.089855+0000","last_became_active":"2023-03-03T06:18:05.775728+0000","last_became_peered":"2023-03-03T06:18:05.775728+0000","last_unstale":"2023-03-03T06:18:07.089855+0000","last_undegraded":"2023-03-03T06:18:07.089855+0000","last_fullsized":"2023-03-03T06:18:07.089855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:17.287057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087740+0000","last_change":"2023-03-03T06:18:05.378086+0000","last_active":"2023-03-03T06:18:07.087740+0000","last_peered":"2023-03-03T06:18:07.087740+0000","last_clean":"2023-03-03T06:18:07.087740+0000","last_became_active":"2023-03-03T06:18:05.377265+0000","last_became_peered":"2023-03-03T06:18:05.377265+0000","last_unstale":"2023-03-03T06:18:07.087740+0000","last_undegraded":"2023-03-03T06:18:07.087740+0000","last_fullsized":"2023-03-03T06:18:07.087740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761128+0000","last_change":"2023-03-03T06:18:05.778565+0000","last_active":"2023-03-03T06:18:06.761128+0000","last_peered":"2023-03-03T06:18:06.761128+0000","last_clean":"2023-03-03T06:18:06.761128+0000","last_became_active":"2023-03-03T06:18:05.778231+0000","last_became_peered":"2023-03-03T06:18:05.778231+0000","last_unstale":"2023-03-03T06:18:06.761128+0000","last_undegraded":"2023-03-03T06:18:06.761128+0000","last_fullsized":"2023-03-03T06:18:06.761128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767346+0000","last_change":"2023-03-03T06:18:05.370256+0000","last_active":"2023-03-03T06:18:06.767346+0000","last_peered":"2023-03-03T06:18:06.767346+0000","last_clean":"2023-03-03T06:18:06.767346+0000","last_became_active":"2023-03-03T06:18:05.369970+0000","last_became_peered":"2023-03-03T06:18:05.369970+0000","last_unstale":"2023-03-03T06:18:06.767346+0000","last_undegraded":"2023-03-03T06:18:06.767346+0000","last_fullsized":"2023-03-03T06:18:06.767346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761149+0000","last_change":"2023-03-03T06:18:05.766019+0000","last_active":"2023-03-03T06:18:06.761149+0000","last_peered":"2023-03-03T06:18:06.761149+0000","last_clean":"2023-03-03T06:18:06.761149+0000","last_became_active":"2023-03-03T06:18:05.765768+0000","last_became_peered":"2023-03-03T06:18:05.765768+0000","last_unstale":"2023-03-03T06:18:06.761149+0000","last_undegraded":"2023-03-03T06:18:06.761149+0000","last_fullsized":"2023-03-03T06:18:06.761149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766991+0000","last_change":"2023-03-03T06:18:05.367491+0000","last_active":"2023-03-03T06:18:06.766991+0000","last_peered":"2023-03-03T06:18:06.766991+0000","last_clean":"2023-03-03T06:18:06.766991+0000","last_became_active":"2023-03-03T06:18:05.367177+0000","last_became_peered":"2023-03-03T06:18:05.367177+0000","last_unstale":"2023-03-03T06:18:06.766991+0000","last_undegraded":"2023-03-03T06:18:06.766991+0000","last_fullsized":"2023-03-03T06:18:06.766991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:15:55.267658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086167+0000","last_change":"2023-03-03T06:18:05.778585+0000","last_active":"2023-03-03T06:18:07.086167+0000","last_peered":"2023-03-03T06:18:07.086167+0000","last_clean":"2023-03-03T06:18:07.086167+0000","last_became_active":"2023-03-03T06:18:05.777053+0000","last_became_peered":"2023-03-03T06:18:05.777053+0000","last_unstale":"2023-03-03T06:18:07.086167+0000","last_undegraded":"2023-03-03T06:18:07.086167+0000","last_fullsized":"2023-03-03T06:18:07.086167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088203+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:07.088203+0000","last_peered":"2023-03-03T06:18:07.088203+0000","last_clean":"2023-03-03T06:18:07.088203+0000","last_became_active":"2023-03-03T06:18:05.372051+0000","last_became_peered":"2023-03-03T06:18:05.372051+0000","last_unstale":"2023-03-03T06:18:07.088203+0000","last_undegraded":"2023-03-03T06:18:07.088203+0000","last_fullsized":"2023-03-03T06:18:07.088203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088167+0000","last_change":"2023-03-03T06:18:05.779300+0000","last_active":"2023-03-03T06:18:07.088167+0000","last_peered":"2023-03-03T06:18:07.088167+0000","last_clean":"2023-03-03T06:18:07.088167+0000","last_became_active":"2023-03-03T06:18:05.778878+0000","last_became_peered":"2023-03-03T06:18:05.778878+0000","last_unstale":"2023-03-03T06:18:07.088167+0000","last_undegraded":"2023-03-03T06:18:07.088167+0000","last_fullsized":"2023-03-03T06:18:07.088167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766613+0000","last_change":"2023-03-03T06:18:05.377498+0000","last_active":"2023-03-03T06:18:06.766613+0000","last_peered":"2023-03-03T06:18:06.766613+0000","last_clean":"2023-03-03T06:18:06.766613+0000","last_became_active":"2023-03-03T06:18:05.377168+0000","last_became_peered":"2023-03-03T06:18:05.377168+0000","last_unstale":"2023-03-03T06:18:06.766613+0000","last_undegraded":"2023-03-03T06:18:06.766613+0000","last_fullsized":"2023-03-03T06:18:06.766613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:38.382151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761390+0000","last_change":"2023-03-03T06:18:05.779094+0000","last_active":"2023-03-03T06:18:06.761390+0000","last_peered":"2023-03-03T06:18:06.761390+0000","last_clean":"2023-03-03T06:18:06.761390+0000","last_became_active":"2023-03-03T06:18:05.778802+0000","last_became_peered":"2023-03-03T06:18:05.778802+0000","last_unstale":"2023-03-03T06:18:06.761390+0000","last_undegraded":"2023-03-03T06:18:06.761390+0000","last_fullsized":"2023-03-03T06:18:06.761390+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087593+0000","last_change":"2023-03-03T06:18:05.376181+0000","last_active":"2023-03-03T06:18:07.087593+0000","last_peered":"2023-03-03T06:18:07.087593+0000","last_clean":"2023-03-03T06:18:07.087593+0000","last_became_active":"2023-03-03T06:18:05.375857+0000","last_became_peered":"2023-03-03T06:18:05.375857+0000","last_unstale":"2023-03-03T06:18:07.087593+0000","last_undegraded":"2023-03-03T06:18:07.087593+0000","last_fullsized":"2023-03-03T06:18:07.087593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767254+0000","last_change":"2023-03-03T06:18:05.775649+0000","last_active":"2023-03-03T06:18:06.767254+0000","last_peered":"2023-03-03T06:18:06.767254+0000","last_clean":"2023-03-03T06:18:06.767254+0000","last_became_active":"2023-03-03T06:18:05.775370+0000","last_became_peered":"2023-03-03T06:18:05.775370+0000","last_unstale":"2023-03-03T06:18:06.767254+0000","last_undegraded":"2023-03-03T06:18:06.767254+0000","last_fullsized":"2023-03-03T06:18:06.767254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766285+0000","last_change":"2023-03-03T06:18:05.377478+0000","last_active":"2023-03-03T06:18:06.766285+0000","last_peered":"2023-03-03T06:18:06.766285+0000","last_clean":"2023-03-03T06:18:06.766285+0000","last_became_active":"2023-03-03T06:18:05.377177+0000","last_became_peered":"2023-03-03T06:18:05.377177+0000","last_unstale":"2023-03-03T06:18:06.766285+0000","last_undegraded":"2023-03-03T06:18:06.766285+0000","last_fullsized":"2023-03-03T06:18:06.766285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:00:42.138865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087675+0000","last_change":"2023-03-03T06:18:05.771983+0000","last_active":"2023-03-03T06:18:07.087675+0000","last_peered":"2023-03-03T06:18:07.087675+0000","last_clean":"2023-03-03T06:18:07.087675+0000","last_became_active":"2023-03-03T06:18:05.771526+0000","last_became_peered":"2023-03-03T06:18:05.771526+0000","last_unstale":"2023-03-03T06:18:07.087675+0000","last_undegraded":"2023-03-03T06:18:07.087675+0000","last_fullsized":"2023-03-03T06:18:07.087675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089756+0000","last_change":"2023-03-03T06:18:05.379123+0000","last_active":"2023-03-03T06:18:07.089756+0000","last_peered":"2023-03-03T06:18:07.089756+0000","last_clean":"2023-03-03T06:18:07.089756+0000","last_became_active":"2023-03-03T06:18:05.378826+0000","last_became_peered":"2023-03-03T06:18:05.378826+0000","last_unstale":"2023-03-03T06:18:07.089756+0000","last_undegraded":"2023-03-03T06:18:07.089756+0000","last_fullsized":"2023-03-03T06:18:07.089756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761645+0000","last_change":"2023-03-03T06:18:05.766998+0000","last_active":"2023-03-03T06:18:06.761645+0000","last_peered":"2023-03-03T06:18:06.761645+0000","last_clean":"2023-03-03T06:18:06.761645+0000","last_became_active":"2023-03-03T06:18:05.764785+0000","last_became_peered":"2023-03-03T06:18:05.764785+0000","last_unstale":"2023-03-03T06:18:06.761645+0000","last_undegraded":"2023-03-03T06:18:06.761645+0000","last_fullsized":"2023-03-03T06:18:06.761645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:37:32.599371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767168+0000","last_change":"2023-03-03T06:18:05.377431+0000","last_active":"2023-03-03T06:18:06.767168+0000","last_peered":"2023-03-03T06:18:06.767168+0000","last_clean":"2023-03-03T06:18:06.767168+0000","last_became_active":"2023-03-03T06:18:05.377107+0000","last_became_peered":"2023-03-03T06:18:05.377107+0000","last_unstale":"2023-03-03T06:18:06.767168+0000","last_undegraded":"2023-03-03T06:18:06.767168+0000","last_fullsized":"2023-03-03T06:18:06.767168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766647+0000","last_change":"2023-03-03T06:18:05.774893+0000","last_active":"2023-03-03T06:18:06.766647+0000","last_peered":"2023-03-03T06:18:06.766647+0000","last_clean":"2023-03-03T06:18:06.766647+0000","last_became_active":"2023-03-03T06:18:05.774256+0000","last_became_peered":"2023-03-03T06:18:05.774256+0000","last_unstale":"2023-03-03T06:18:06.766647+0000","last_undegraded":"2023-03-03T06:18:06.766647+0000","last_fullsized":"2023-03-03T06:18:06.766647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766744+0000","last_change":"2023-03-03T06:18:05.377497+0000","last_active":"2023-03-03T06:18:06.766744+0000","last_peered":"2023-03-03T06:18:06.766744+0000","last_clean":"2023-03-03T06:18:06.766744+0000","last_became_active":"2023-03-03T06:18:05.377215+0000","last_became_peered":"2023-03-03T06:18:05.377215+0000","last_unstale":"2023-03-03T06:18:06.766744+0000","last_undegraded":"2023-03-03T06:18:06.766744+0000","last_fullsized":"2023-03-03T06:18:06.766744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088051+0000","last_change":"2023-03-03T06:18:05.781538+0000","last_active":"2023-03-03T06:18:07.088051+0000","last_peered":"2023-03-03T06:18:07.088051+0000","last_clean":"2023-03-03T06:18:07.088051+0000","last_became_active":"2023-03-03T06:18:05.781285+0000","last_became_peered":"2023-03-03T06:18:05.781285+0000","last_unstale":"2023-03-03T06:18:07.088051+0000","last_undegraded":"2023-03-03T06:18:07.088051+0000","last_fullsized":"2023-03-03T06:18:07.088051+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:45:41.942395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767086+0000","last_change":"2023-03-03T06:18:05.369584+0000","last_active":"2023-03-03T06:18:06.767086+0000","last_peered":"2023-03-03T06:18:06.767086+0000","last_clean":"2023-03-03T06:18:06.767086+0000","last_became_active":"2023-03-03T06:18:05.368904+0000","last_became_peered":"2023-03-03T06:18:05.368904+0000","last_unstale":"2023-03-03T06:18:06.767086+0000","last_undegraded":"2023-03-03T06:18:06.767086+0000","last_fullsized":"2023-03-03T06:18:06.767086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:23.861622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089809+0000","last_change":"2023-03-03T06:18:05.779823+0000","last_active":"2023-03-03T06:18:07.089809+0000","last_peered":"2023-03-03T06:18:07.089809+0000","last_clean":"2023-03-03T06:18:07.089809+0000","last_became_active":"2023-03-03T06:18:05.779561+0000","last_became_peered":"2023-03-03T06:18:05.779561+0000","last_unstale":"2023-03-03T06:18:07.089809+0000","last_undegraded":"2023-03-03T06:18:07.089809+0000","last_fullsized":"2023-03-03T06:18:07.089809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:54.968231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767196+0000","last_change":"2023-03-03T06:18:05.372345+0000","last_active":"2023-03-03T06:18:06.767196+0000","last_peered":"2023-03-03T06:18:06.767196+0000","last_clean":"2023-03-03T06:18:06.767196+0000","last_became_active":"2023-03-03T06:18:05.372057+0000","last_became_peered":"2023-03-03T06:18:05.372057+0000","last_unstale":"2023-03-03T06:18:06.767196+0000","last_undegraded":"2023-03-03T06:18:06.767196+0000","last_fullsized":"2023-03-03T06:18:06.767196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:21:53.962695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087624+0000","last_change":"2023-03-03T06:18:05.780035+0000","last_active":"2023-03-03T06:18:07.087624+0000","last_peered":"2023-03-03T06:18:07.087624+0000","last_clean":"2023-03-03T06:18:07.087624+0000","last_became_active":"2023-03-03T06:18:05.779650+0000","last_became_peered":"2023-03-03T06:18:05.779650+0000","last_unstale":"2023-03-03T06:18:07.087624+0000","last_undegraded":"2023-03-03T06:18:07.087624+0000","last_fullsized":"2023-03-03T06:18:07.087624+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761617+0000","last_change":"2023-03-03T06:18:05.379968+0000","last_active":"2023-03-03T06:18:06.761617+0000","last_peered":"2023-03-03T06:18:06.761617+0000","last_clean":"2023-03-03T06:18:06.761617+0000","last_became_active":"2023-03-03T06:18:05.379635+0000","last_became_peered":"2023-03-03T06:18:05.379635+0000","last_unstale":"2023-03-03T06:18:06.761617+0000","last_undegraded":"2023-03-03T06:18:06.761617+0000","last_fullsized":"2023-03-03T06:18:06.761617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088190+0000","last_change":"2023-03-03T06:18:05.768972+0000","last_active":"2023-03-03T06:18:07.088190+0000","last_peered":"2023-03-03T06:18:07.088190+0000","last_clean":"2023-03-03T06:18:07.088190+0000","last_became_active":"2023-03-03T06:18:05.768647+0000","last_became_peered":"2023-03-03T06:18:05.768647+0000","last_unstale":"2023-03-03T06:18:07.088190+0000","last_undegraded":"2023-03-03T06:18:07.088190+0000","last_fullsized":"2023-03-03T06:18:07.088190+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:15:35.033362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089794+0000","last_change":"2023-03-03T06:18:05.378643+0000","last_active":"2023-03-03T06:18:07.089794+0000","last_peered":"2023-03-03T06:18:07.089794+0000","last_clean":"2023-03-03T06:18:07.089794+0000","last_became_active":"2023-03-03T06:18:05.378333+0000","last_became_peered":"2023-03-03T06:18:05.378333+0000","last_unstale":"2023-03-03T06:18:07.089794+0000","last_undegraded":"2023-03-03T06:18:07.089794+0000","last_fullsized":"2023-03-03T06:18:07.089794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088249+0000","last_change":"2023-03-03T06:18:05.774576+0000","last_active":"2023-03-03T06:18:07.088249+0000","last_peered":"2023-03-03T06:18:07.088249+0000","last_clean":"2023-03-03T06:18:07.088249+0000","last_became_active":"2023-03-03T06:18:05.774278+0000","last_became_peered":"2023-03-03T06:18:05.774278+0000","last_unstale":"2023-03-03T06:18:07.088249+0000","last_undegraded":"2023-03-03T06:18:07.088249+0000","last_fullsized":"2023-03-03T06:18:07.088249+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:20.212207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088073+0000","last_change":"2023-03-03T06:18:05.374483+0000","last_active":"2023-03-03T06:18:07.088073+0000","last_peered":"2023-03-03T06:18:07.088073+0000","last_clean":"2023-03-03T06:18:07.088073+0000","last_became_active":"2023-03-03T06:18:05.374224+0000","last_became_peered":"2023-03-03T06:18:05.374224+0000","last_unstale":"2023-03-03T06:18:07.088073+0000","last_undegraded":"2023-03-03T06:18:07.088073+0000","last_fullsized":"2023-03-03T06:18:07.088073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088262+0000","last_change":"2023-03-03T06:18:05.779652+0000","last_active":"2023-03-03T06:18:07.088262+0000","last_peered":"2023-03-03T06:18:07.088262+0000","last_clean":"2023-03-03T06:18:07.088262+0000","last_became_active":"2023-03-03T06:18:05.778557+0000","last_became_peered":"2023-03-03T06:18:05.778557+0000","last_unstale":"2023-03-03T06:18:07.088262+0000","last_undegraded":"2023-03-03T06:18:07.088262+0000","last_fullsized":"2023-03-03T06:18:07.088262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761087+0000","last_change":"2023-03-03T06:18:05.373177+0000","last_active":"2023-03-03T06:18:06.761087+0000","last_peered":"2023-03-03T06:18:06.761087+0000","last_clean":"2023-03-03T06:18:06.761087+0000","last_became_active":"2023-03-03T06:18:05.372879+0000","last_became_peered":"2023-03-03T06:18:05.372879+0000","last_unstale":"2023-03-03T06:18:06.761087+0000","last_undegraded":"2023-03-03T06:18:06.761087+0000","last_fullsized":"2023-03-03T06:18:06.761087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086271+0000","last_change":"2023-03-03T06:18:05.778960+0000","last_active":"2023-03-03T06:18:07.086271+0000","last_peered":"2023-03-03T06:18:07.086271+0000","last_clean":"2023-03-03T06:18:07.086271+0000","last_became_active":"2023-03-03T06:18:05.777570+0000","last_became_peered":"2023-03-03T06:18:05.777570+0000","last_unstale":"2023-03-03T06:18:07.086271+0000","last_undegraded":"2023-03-03T06:18:07.086271+0000","last_fullsized":"2023-03-03T06:18:07.086271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:43.532127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.775376+0000","last_change":"2023-03-03T06:18:00.701360+0000","last_active":"2023-03-03T06:18:05.775376+0000","last_peered":"2023-03-03T06:18:05.775376+0000","last_clean":"2023-03-03T06:18:05.775376+0000","last_became_active":"2023-03-03T06:18:00.701042+0000","last_became_peered":"2023-03-03T06:18:00.701042+0000","last_unstale":"2023-03-03T06:18:05.775376+0000","last_undegraded":"2023-03-03T06:18:05.775376+0000","last_fullsized":"2023-03-03T06:18:05.775376+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:10.253889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767335+0000","last_change":"2023-03-03T06:18:05.372375+0000","last_active":"2023-03-03T06:18:06.767335+0000","last_peered":"2023-03-03T06:18:06.767335+0000","last_clean":"2023-03-03T06:18:06.767335+0000","last_became_active":"2023-03-03T06:18:05.371680+0000","last_became_peered":"2023-03-03T06:18:05.371680+0000","last_unstale":"2023-03-03T06:18:06.767335+0000","last_undegraded":"2023-03-03T06:18:06.767335+0000","last_fullsized":"2023-03-03T06:18:06.767335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:45.912038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086177+0000","last_change":"2023-03-03T06:18:05.778931+0000","last_active":"2023-03-03T06:18:07.086177+0000","last_peered":"2023-03-03T06:18:07.086177+0000","last_clean":"2023-03-03T06:18:07.086177+0000","last_became_active":"2023-03-03T06:18:05.778723+0000","last_became_peered":"2023-03-03T06:18:05.778723+0000","last_unstale":"2023-03-03T06:18:07.086177+0000","last_undegraded":"2023-03-03T06:18:07.086177+0000","last_fullsized":"2023-03-03T06:18:07.086177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:47:08.350712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.764427+0000","last_change":"2023-03-03T06:18:00.351410+0000","last_active":"2023-03-03T06:18:05.764427+0000","last_peered":"2023-03-03T06:18:05.764427+0000","last_clean":"2023-03-03T06:18:05.764427+0000","last_became_active":"2023-03-03T06:18:00.351036+0000","last_became_peered":"2023-03-03T06:18:00.351036+0000","last_unstale":"2023-03-03T06:18:05.764427+0000","last_undegraded":"2023-03-03T06:18:05.764427+0000","last_fullsized":"2023-03-03T06:18:05.764427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088156+0000","last_change":"2023-03-03T06:18:05.378884+0000","last_active":"2023-03-03T06:18:07.088156+0000","last_peered":"2023-03-03T06:18:07.088156+0000","last_clean":"2023-03-03T06:18:07.088156+0000","last_became_active":"2023-03-03T06:18:05.378597+0000","last_became_peered":"2023-03-03T06:18:05.378597+0000","last_unstale":"2023-03-03T06:18:07.088156+0000","last_undegraded":"2023-03-03T06:18:07.088156+0000","last_fullsized":"2023-03-03T06:18:07.088156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086204+0000","last_change":"2023-03-03T06:18:05.777398+0000","last_active":"2023-03-03T06:18:07.086204+0000","last_peered":"2023-03-03T06:18:07.086204+0000","last_clean":"2023-03-03T06:18:07.086204+0000","last_became_active":"2023-03-03T06:18:05.777065+0000","last_became_peered":"2023-03-03T06:18:05.777065+0000","last_unstale":"2023-03-03T06:18:07.086204+0000","last_undegraded":"2023-03-03T06:18:07.086204+0000","last_fullsized":"2023-03-03T06:18:07.086204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.760094+0000","last_change":"2023-03-03T06:18:00.351233+0000","last_active":"2023-03-03T06:18:05.760094+0000","last_peered":"2023-03-03T06:18:05.760094+0000","last_clean":"2023-03-03T06:18:05.760094+0000","last_became_active":"2023-03-03T06:18:00.350944+0000","last_became_peered":"2023-03-03T06:18:00.350944+0000","last_unstale":"2023-03-03T06:18:05.760094+0000","last_undegraded":"2023-03-03T06:18:05.760094+0000","last_fullsized":"2023-03-03T06:18:05.760094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:08:23.569811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086137+0000","last_change":"2023-03-03T06:18:05.374876+0000","last_active":"2023-03-03T06:18:07.086137+0000","last_peered":"2023-03-03T06:18:07.086137+0000","last_clean":"2023-03-03T06:18:07.086137+0000","last_became_active":"2023-03-03T06:18:05.374266+0000","last_became_peered":"2023-03-03T06:18:05.374266+0000","last_unstale":"2023-03-03T06:18:07.086137+0000","last_undegraded":"2023-03-03T06:18:07.086137+0000","last_fullsized":"2023-03-03T06:18:07.086137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086261+0000","last_change":"2023-03-03T06:18:05.779139+0000","last_active":"2023-03-03T06:18:07.086261+0000","last_peered":"2023-03-03T06:18:07.086261+0000","last_clean":"2023-03-03T06:18:07.086261+0000","last_became_active":"2023-03-03T06:18:05.777315+0000","last_became_peered":"2023-03-03T06:18:05.777315+0000","last_unstale":"2023-03-03T06:18:07.086261+0000","last_undegraded":"2023-03-03T06:18:07.086261+0000","last_fullsized":"2023-03-03T06:18:07.086261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.762243+0000","last_change":"2023-03-03T06:18:00.351257+0000","last_active":"2023-03-03T06:18:05.762243+0000","last_peered":"2023-03-03T06:18:05.762243+0000","last_clean":"2023-03-03T06:18:05.762243+0000","last_became_active":"2023-03-03T06:18:00.350933+0000","last_became_peered":"2023-03-03T06:18:00.350933+0000","last_unstale":"2023-03-03T06:18:05.762243+0000","last_undegraded":"2023-03-03T06:18:05.762243+0000","last_fullsized":"2023-03-03T06:18:05.762243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758385+0000","last_change":"2023-03-03T06:17:54.307666+0000","last_active":"2023-03-03T06:18:05.758385+0000","last_peered":"2023-03-03T06:18:05.758385+0000","last_clean":"2023-03-03T06:18:05.758385+0000","last_became_active":"2023-03-03T06:17:54.306552+0000","last_became_peered":"2023-03-03T06:17:54.306552+0000","last_unstale":"2023-03-03T06:18:05.758385+0000","last_undegraded":"2023-03-03T06:18:05.758385+0000","last_fullsized":"2023-03-03T06:18:05.758385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:53.091715+0000","last_clean_scrub_stamp":"2023-03-03T06:17:53.091715+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:56:41.267904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761267+0000","last_change":"2023-03-03T06:18:05.366688+0000","last_active":"2023-03-03T06:18:06.761267+0000","last_peered":"2023-03-03T06:18:06.761267+0000","last_clean":"2023-03-03T06:18:06.761267+0000","last_became_active":"2023-03-03T06:18:05.366412+0000","last_became_peered":"2023-03-03T06:18:05.366412+0000","last_unstale":"2023-03-03T06:18:06.761267+0000","last_undegraded":"2023-03-03T06:18:06.761267+0000","last_fullsized":"2023-03-03T06:18:06.761267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089833+0000","last_change":"2023-03-03T06:18:05.781486+0000","last_active":"2023-03-03T06:18:07.089833+0000","last_peered":"2023-03-03T06:18:07.089833+0000","last_clean":"2023-03-03T06:18:07.089833+0000","last_became_active":"2023-03-03T06:18:05.781220+0000","last_became_peered":"2023-03-03T06:18:05.781220+0000","last_unstale":"2023-03-03T06:18:07.089833+0000","last_undegraded":"2023-03-03T06:18:07.089833+0000","last_fullsized":"2023-03-03T06:18:07.089833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:09:39.387687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.758932+0000","last_change":"2023-03-03T06:18:00.345967+0000","last_active":"2023-03-03T06:18:05.758932+0000","last_peered":"2023-03-03T06:18:05.758932+0000","last_clean":"2023-03-03T06:18:05.758932+0000","last_became_active":"2023-03-03T06:18:00.345666+0000","last_became_peered":"2023-03-03T06:18:00.345666+0000","last_unstale":"2023-03-03T06:18:05.758932+0000","last_undegraded":"2023-03-03T06:18:05.758932+0000","last_fullsized":"2023-03-03T06:18:05.758932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:04:22.933314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087574+0000","last_change":"2023-03-03T06:18:05.379214+0000","last_active":"2023-03-03T06:18:07.087574+0000","last_peered":"2023-03-03T06:18:07.087574+0000","last_clean":"2023-03-03T06:18:07.087574+0000","last_became_active":"2023-03-03T06:18:05.378867+0000","last_became_peered":"2023-03-03T06:18:05.378867+0000","last_unstale":"2023-03-03T06:18:07.087574+0000","last_undegraded":"2023-03-03T06:18:07.087574+0000","last_fullsized":"2023-03-03T06:18:07.087574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:54:16.597676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087711+0000","last_change":"2023-03-03T06:18:05.782275+0000","last_active":"2023-03-03T06:18:07.087711+0000","last_peered":"2023-03-03T06:18:07.087711+0000","last_clean":"2023-03-03T06:18:07.087711+0000","last_became_active":"2023-03-03T06:18:05.781968+0000","last_became_peered":"2023-03-03T06:18:05.781968+0000","last_unstale":"2023-03-03T06:18:07.087711+0000","last_undegraded":"2023-03-03T06:18:07.087711+0000","last_fullsized":"2023-03-03T06:18:07.087711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.759159+0000","last_change":"2023-03-03T06:18:00.701652+0000","last_active":"2023-03-03T06:18:05.759159+0000","last_peered":"2023-03-03T06:18:05.759159+0000","last_clean":"2023-03-03T06:18:05.759159+0000","last_became_active":"2023-03-03T06:18:00.701382+0000","last_became_peered":"2023-03-03T06:18:00.701382+0000","last_unstale":"2023-03-03T06:18:05.759159+0000","last_undegraded":"2023-03-03T06:18:05.759159+0000","last_fullsized":"2023-03-03T06:18:05.759159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:50:16.060640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086214+0000","last_change":"2023-03-03T06:18:05.374785+0000","last_active":"2023-03-03T06:18:07.086214+0000","last_peered":"2023-03-03T06:18:07.086214+0000","last_clean":"2023-03-03T06:18:07.086214+0000","last_became_active":"2023-03-03T06:18:05.374322+0000","last_became_peered":"2023-03-03T06:18:05.374322+0000","last_unstale":"2023-03-03T06:18:07.086214+0000","last_undegraded":"2023-03-03T06:18:07.086214+0000","last_fullsized":"2023-03-03T06:18:07.086214+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:55:42.194773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087664+0000","last_change":"2023-03-03T06:18:05.775603+0000","last_active":"2023-03-03T06:18:07.087664+0000","last_peered":"2023-03-03T06:18:07.087664+0000","last_clean":"2023-03-03T06:18:07.087664+0000","last_became_active":"2023-03-03T06:18:05.775350+0000","last_became_peered":"2023-03-03T06:18:05.775350+0000","last_unstale":"2023-03-03T06:18:07.087664+0000","last_undegraded":"2023-03-03T06:18:07.087664+0000","last_fullsized":"2023-03-03T06:18:07.087664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:39:57.779927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761609+0000","last_change":"2023-03-03T06:18:05.371388+0000","last_active":"2023-03-03T06:18:06.761609+0000","last_peered":"2023-03-03T06:18:06.761609+0000","last_clean":"2023-03-03T06:18:06.761609+0000","last_became_active":"2023-03-03T06:18:05.371026+0000","last_became_peered":"2023-03-03T06:18:05.371026+0000","last_unstale":"2023-03-03T06:18:06.761609+0000","last_undegraded":"2023-03-03T06:18:06.761609+0000","last_fullsized":"2023-03-03T06:18:06.761609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:54:16.196146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.769697+0000","last_change":"2023-03-03T06:18:00.349337+0000","last_active":"2023-03-03T06:18:05.769697+0000","last_peered":"2023-03-03T06:18:05.769697+0000","last_clean":"2023-03-03T06:18:05.769697+0000","last_became_active":"2023-03-03T06:18:00.349101+0000","last_became_peered":"2023-03-03T06:18:00.349101+0000","last_unstale":"2023-03-03T06:18:05.769697+0000","last_undegraded":"2023-03-03T06:18:05.769697+0000","last_fullsized":"2023-03-03T06:18:05.769697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:13.472474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761022+0000","last_change":"2023-03-03T06:18:05.762420+0000","last_active":"2023-03-03T06:18:06.761022+0000","last_peered":"2023-03-03T06:18:06.761022+0000","last_clean":"2023-03-03T06:18:06.761022+0000","last_became_active":"2023-03-03T06:18:05.760911+0000","last_became_peered":"2023-03-03T06:18:05.760911+0000","last_unstale":"2023-03-03T06:18:06.761022+0000","last_undegraded":"2023-03-03T06:18:06.761022+0000","last_fullsized":"2023-03-03T06:18:06.761022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:35.723202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:05.766068+0000","last_change":"2023-03-03T06:18:00.352230+0000","last_active":"2023-03-03T06:18:05.766068+0000","last_peered":"2023-03-03T06:18:05.766068+0000","last_clean":"2023-03-03T06:18:05.766068+0000","last_became_active":"2023-03-03T06:18:00.351881+0000","last_became_peered":"2023-03-03T06:18:00.351881+0000","last_unstale":"2023-03-03T06:18:05.766068+0000","last_undegraded":"2023-03-03T06:18:05.766068+0000","last_fullsized":"2023-03-03T06:18:05.766068+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:17:59.321133+0000","last_clean_scrub_stamp":"2023-03-03T06:17:59.321133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:18:24.984007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.567915+0000","last_change":"2023-03-03T06:18:05.382451+0000","last_active":"2023-03-03T06:18:07.567915+0000","last_peered":"2023-03-03T06:18:07.567915+0000","last_clean":"2023-03-03T06:18:07.567915+0000","last_became_active":"2023-03-03T06:18:05.382112+0000","last_became_peered":"2023-03-03T06:18:05.382112+0000","last_unstale":"2023-03-03T06:18:07.567915+0000","last_undegraded":"2023-03-03T06:18:07.567915+0000","last_fullsized":"2023-03-03T06:18:07.567915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:47:57.435210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088062+0000","last_change":"2023-03-03T06:18:05.777771+0000","last_active":"2023-03-03T06:18:07.088062+0000","last_peered":"2023-03-03T06:18:07.088062+0000","last_clean":"2023-03-03T06:18:07.088062+0000","last_became_active":"2023-03-03T06:18:05.777457+0000","last_became_peered":"2023-03-03T06:18:05.777457+0000","last_unstale":"2023-03-03T06:18:07.088062+0000","last_undegraded":"2023-03-03T06:18:07.088062+0000","last_fullsized":"2023-03-03T06:18:07.088062+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T07:25:24.867158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087635+0000","last_change":"2023-03-03T06:18:05.382340+0000","last_active":"2023-03-03T06:18:07.087635+0000","last_peered":"2023-03-03T06:18:07.087635+0000","last_clean":"2023-03-03T06:18:07.087635+0000","last_became_active":"2023-03-03T06:18:05.382051+0000","last_became_peered":"2023-03-03T06:18:05.382051+0000","last_unstale":"2023-03-03T06:18:07.087635+0000","last_undegraded":"2023-03-03T06:18:07.087635+0000","last_fullsized":"2023-03-03T06:18:07.087635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:38.701961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089777+0000","last_change":"2023-03-03T06:18:05.778551+0000","last_active":"2023-03-03T06:18:07.089777+0000","last_peered":"2023-03-03T06:18:07.089777+0000","last_clean":"2023-03-03T06:18:07.089777+0000","last_became_active":"2023-03-03T06:18:05.778166+0000","last_became_peered":"2023-03-03T06:18:05.778166+0000","last_unstale":"2023-03-03T06:18:07.089777+0000","last_undegraded":"2023-03-03T06:18:07.089777+0000","last_fullsized":"2023-03-03T06:18:07.089777+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:33:39.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.766897+0000","last_change":"2023-03-03T06:18:05.371014+0000","last_active":"2023-03-03T06:18:06.766897+0000","last_peered":"2023-03-03T06:18:06.766897+0000","last_clean":"2023-03-03T06:18:06.766897+0000","last_became_active":"2023-03-03T06:18:05.370730+0000","last_became_peered":"2023-03-03T06:18:05.370730+0000","last_unstale":"2023-03-03T06:18:06.766897+0000","last_undegraded":"2023-03-03T06:18:06.766897+0000","last_fullsized":"2023-03-03T06:18:06.766897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088144+0000","last_change":"2023-03-03T06:18:05.772599+0000","last_active":"2023-03-03T06:18:07.088144+0000","last_peered":"2023-03-03T06:18:07.088144+0000","last_clean":"2023-03-03T06:18:07.088144+0000","last_became_active":"2023-03-03T06:18:05.772305+0000","last_became_peered":"2023-03-03T06:18:05.772305+0000","last_unstale":"2023-03-03T06:18:07.088144+0000","last_undegraded":"2023-03-03T06:18:07.088144+0000","last_fullsized":"2023-03-03T06:18:07.088144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:14.254976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.767411+0000","last_change":"2023-03-03T06:18:05.374466+0000","last_active":"2023-03-03T06:18:06.767411+0000","last_peered":"2023-03-03T06:18:06.767411+0000","last_clean":"2023-03-03T06:18:06.767411+0000","last_became_active":"2023-03-03T06:18:05.374194+0000","last_became_peered":"2023-03-03T06:18:05.374194+0000","last_unstale":"2023-03-03T06:18:06.767411+0000","last_undegraded":"2023-03-03T06:18:06.767411+0000","last_fullsized":"2023-03-03T06:18:06.767411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:00.945489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088131+0000","last_change":"2023-03-03T06:18:05.773340+0000","last_active":"2023-03-03T06:18:07.088131+0000","last_peered":"2023-03-03T06:18:07.088131+0000","last_clean":"2023-03-03T06:18:07.088131+0000","last_became_active":"2023-03-03T06:18:05.772391+0000","last_became_peered":"2023-03-03T06:18:05.772391+0000","last_unstale":"2023-03-03T06:18:07.088131+0000","last_undegraded":"2023-03-03T06:18:07.088131+0000","last_fullsized":"2023-03-03T06:18:07.088131+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:38.014451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.762090+0000","last_change":"2023-03-03T06:18:05.369911+0000","last_active":"2023-03-03T06:18:06.762090+0000","last_peered":"2023-03-03T06:18:06.762090+0000","last_clean":"2023-03-03T06:18:06.762090+0000","last_became_active":"2023-03-03T06:18:05.369605+0000","last_became_peered":"2023-03-03T06:18:05.369605+0000","last_unstale":"2023-03-03T06:18:06.762090+0000","last_undegraded":"2023-03-03T06:18:06.762090+0000","last_fullsized":"2023-03-03T06:18:06.762090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:44:16.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761116+0000","last_change":"2023-03-03T06:18:05.766192+0000","last_active":"2023-03-03T06:18:06.761116+0000","last_peered":"2023-03-03T06:18:06.761116+0000","last_clean":"2023-03-03T06:18:06.761116+0000","last_became_active":"2023-03-03T06:18:05.764744+0000","last_became_peered":"2023-03-03T06:18:05.764744+0000","last_unstale":"2023-03-03T06:18:06.761116+0000","last_undegraded":"2023-03-03T06:18:06.761116+0000","last_fullsized":"2023-03-03T06:18:06.761116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:07:02.001543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.087752+0000","last_change":"2023-03-03T06:18:05.378275+0000","last_active":"2023-03-03T06:18:07.087752+0000","last_peered":"2023-03-03T06:18:07.087752+0000","last_clean":"2023-03-03T06:18:07.087752+0000","last_became_active":"2023-03-03T06:18:05.377803+0000","last_became_peered":"2023-03-03T06:18:05.377803+0000","last_unstale":"2023-03-03T06:18:07.087752+0000","last_undegraded":"2023-03-03T06:18:07.087752+0000","last_fullsized":"2023-03-03T06:18:07.087752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:04.344243+0000","last_clean_scrub_stamp":"2023-03-03T06:18:04.344243+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:37.541185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088033+0000","last_change":"2023-03-03T06:18:05.770801+0000","last_active":"2023-03-03T06:18:07.088033+0000","last_peered":"2023-03-03T06:18:07.088033+0000","last_clean":"2023-03-03T06:18:07.088033+0000","last_became_active":"2023-03-03T06:18:05.770500+0000","last_became_peered":"2023-03-03T06:18:05.770500+0000","last_unstale":"2023-03-03T06:18:07.088033+0000","last_undegraded":"2023-03-03T06:18:07.088033+0000","last_fullsized":"2023-03-03T06:18:07.088033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:38.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.089779+0000","last_change":"2023-03-03T06:18:05.776954+0000","last_active":"2023-03-03T06:18:07.089779+0000","last_peered":"2023-03-03T06:18:07.089779+0000","last_clean":"2023-03-03T06:18:07.089779+0000","last_became_active":"2023-03-03T06:18:05.776665+0000","last_became_peered":"2023-03-03T06:18:05.776665+0000","last_unstale":"2023-03-03T06:18:07.089779+0000","last_undegraded":"2023-03-03T06:18:07.089779+0000","last_fullsized":"2023-03-03T06:18:07.089779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:46.917574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761055+0000","last_change":"2023-03-03T06:18:05.780029+0000","last_active":"2023-03-03T06:18:06.761055+0000","last_peered":"2023-03-03T06:18:06.761055+0000","last_clean":"2023-03-03T06:18:06.761055+0000","last_became_active":"2023-03-03T06:18:05.779805+0000","last_became_peered":"2023-03-03T06:18:05.779805+0000","last_unstale":"2023-03-03T06:18:06.761055+0000","last_undegraded":"2023-03-03T06:18:06.761055+0000","last_fullsized":"2023-03-03T06:18:06.761055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:01:00.271635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.088099+0000","last_change":"2023-03-03T06:18:05.781886+0000","last_active":"2023-03-03T06:18:07.088099+0000","last_peered":"2023-03-03T06:18:07.088099+0000","last_clean":"2023-03-03T06:18:07.088099+0000","last_became_active":"2023-03-03T06:18:05.780587+0000","last_became_peered":"2023-03-03T06:18:05.780587+0000","last_unstale":"2023-03-03T06:18:07.088099+0000","last_undegraded":"2023-03-03T06:18:07.088099+0000","last_fullsized":"2023-03-03T06:18:07.088099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:02.129741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T06:18:06.761713+0000","last_change":"2023-03-03T06:18:05.760339+0000","last_active":"2023-03-03T06:18:06.761713+0000","last_peered":"2023-03-03T06:18:06.761713+0000","last_clean":"2023-03-03T06:18:06.761713+0000","last_became_active":"2023-03-03T06:18:05.759652+0000","last_became_peered":"2023-03-03T06:18:05.759652+0000","last_unstale":"2023-03-03T06:18:06.761713+0000","last_undegraded":"2023-03-03T06:18:06.761713+0000","last_fullsized":"2023-03-03T06:18:06.761713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:08.504851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T06:18:07.086186+0000","last_change":"2023-03-03T06:18:05.777608+0000","last_active":"2023-03-03T06:18:07.086186+0000","last_peered":"2023-03-03T06:18:07.086186+0000","last_clean":"2023-03-03T06:18:07.086186+0000","last_became_active":"2023-03-03T06:18:05.777310+0000","last_became_peered":"2023-03-03T06:18:05.777310+0000","last_unstale":"2023-03-03T06:18:07.086186+0000","last_undegraded":"2023-03-03T06:18:07.086186+0000","last_fullsized":"2023-03-03T06:18:07.086186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T06:18:05.349779+0000","last_clean_scrub_stamp":"2023-03-03T06:18:05.349779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:34.167726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9108,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362732,"statfs":{"total":96636764160,"available":96627437568,"internally_reserved":0,"allocated":864256,"data_stored":681731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7988,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363852,"statfs":{"total":96636764160,"available":96628584448,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9559999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9949999999999992}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8052,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363788,"statfs":{"total":96636764160,"available":96628518912,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0340000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1659999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.202}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.984999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.061}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9084,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362756,"statfs":{"total":96636764160,"available":96627462144,"internally_reserved":0,"allocated":839680,"data_stored":680533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7924,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363916,"statfs":{"total":96636764160,"available":96628649984,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.834}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.032}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.786}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8056,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363784,"statfs":{"total":96636764160,"available":96628514816,"internally_reserved":0,"allocated":245760,"data_stored":89689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8072,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363768,"statfs":{"total":96636764160,"available":96628498432,"internally_reserved":0,"allocated":262144,"data_stored":91203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7996,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363844,"statfs":{"total":96636764160,"available":96628576256,"internally_reserved":0,"allocated":249856,"data_stored":90235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0339999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T06:18:14.608 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-03-03T06:18:14.608 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-03-03T06:18:14.608 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-03-03T06:18:14.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-03-03T06:18:14.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.759+0000 7f3547c56700 1 Processor -- start 2023-03-03T06:18:14.761 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.759+0000 7f3547c56700 1 -- start start 2023-03-03T06:18:14.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f3547c56700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35400c4aa0 0x7f354006cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f3547c56700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35401506b0 0x7f35400c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f3547c56700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3540152a90 con 0x7f35401531f0 2023-03-03T06:18:14.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f3547c56700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35400c3980 con 0x7f35401506b0 2023-03-03T06:18:14.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f3547c56700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35400c4620 con 0x7f35400c4aa0 2023-03-03T06:18:14.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35401506b0 0x7f35400c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.764 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35401506b0 0x7f35400c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34418/0 (socket says 172.21.15.90:34418) 2023-03-03T06:18:14.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 -- 172.21.15.90:0/1332210743 learned_addr learned my addr 172.21.15.90:0/1332210743 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:14.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35451f1700 1 --2- 172.21.15.90:0/1332210743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35400c4aa0 0x7f354006cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 -- 172.21.15.90:0/1332210743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35400c4aa0 msgr2=0x7f354006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:14.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 --2- 172.21.15.90:0/1332210743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35400c4aa0 0x7f354006cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 -- 172.21.15.90:0/1332210743 >> v1:172.21.15.90:6789/0 conn(0x7f35401531f0 legacy=0x7f354006c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.765 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 -- 172.21.15.90:0/1332210743 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35401527b0 con 0x7f35401506b0 2023-03-03T06:18:14.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35461f3700 1 --2- 172.21.15.90:0/1332210743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35401506b0 0x7f35400c3e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3538009e50 tx=0x7f353800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfc60b78674481d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35449f0700 1 -- 172.21.15.90:0/1332210743 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353800e070 con 0x7f35401506b0 2023-03-03T06:18:14.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35449f0700 1 -- 172.21.15.90:0/1332210743 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353800b9c0 con 0x7f35401506b0 2023-03-03T06:18:14.766 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.760+0000 7f35449f0700 1 -- 172.21.15.90:0/1332210743 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353800bd70 con 0x7f35401506b0 2023-03-03T06:18:14.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 -- 172.21.15.90:0/1332210743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35401506b0 msgr2=0x7f35400c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 --2- 172.21.15.90:0/1332210743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35401506b0 0x7f35400c3e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3538009e50 tx=0x7f353800b330 comp rx=0 tx=0).stop 2023-03-03T06:18:14.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 -- 172.21.15.90:0/1332210743 shutdown_connections 2023-03-03T06:18:14.767 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 --2- 172.21.15.90:0/1332210743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35401506b0 0x7f35400c3e10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 --2- 172.21.15.90:0/1332210743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35400c4aa0 0x7f354006cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 -- 172.21.15.90:0/1332210743 >> 172.21.15.90:0/1332210743 conn(0x7f35400a7060 msgr2=0x7f35400a7460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:14.768 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 -- 172.21.15.90:0/1332210743 shutdown_connections 2023-03-03T06:18:14.769 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.761+0000 7f3547c56700 1 -- 172.21.15.90:0/1332210743 wait complete. 2023-03-03T06:18:14.769 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.762+0000 7f3547c56700 1 Processor -- start 2023-03-03T06:18:14.769 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.762+0000 7f3547c56700 1 -- start start 2023-03-03T06:18:14.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f3547c56700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35400c4aa0 0x7f35400bd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f3547c56700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35401506b0 0x7f35400b7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.774 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f3547c56700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35401531f0 0x7f35401e7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.775 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f3547c56700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35400bcb10 con 0x7f35400c4aa0 2023-03-03T06:18:14.775 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f3547c56700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401e8e70 con 0x7f35401531f0 2023-03-03T06:18:14.775 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f3547c56700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35401e89e0 con 0x7f35401506b0 2023-03-03T06:18:14.775 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35400c4aa0 0x7f35400bd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.776 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35400c4aa0 0x7f35400bd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34420/0 (socket says 172.21.15.90:34420) 2023-03-03T06:18:14.776 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 -- 172.21.15.90:0/922094354 learned_addr learned my addr 172.21.15.90:0/922094354 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:14.776 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35451f1700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35401506b0 0x7f35400b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.776 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35461f3700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35401531f0 0x7f35401e7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.777 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 -- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35401506b0 msgr2=0x7f35400b7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.777 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35401506b0 0x7f35400b7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.777 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 -- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35401531f0 msgr2=0x7f35401e7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.778 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35401531f0 0x7f35401e7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.778 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 -- 172.21.15.90:0/922094354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3538009b30 con 0x7f35400c4aa0 2023-03-03T06:18:14.778 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35461f3700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35401531f0 0x7f35401e7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:14.778 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35459f2700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35400c4aa0 0x7f35400bd6f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c510 tx=0x7f353c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.779 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35367fc700 1 -- 172.21.15.90:0/922094354 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0052d0 con 0x7f35400c4aa0 2023-03-03T06:18:14.779 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35367fc700 1 -- 172.21.15.90:0/922094354 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c005430 con 0x7f35400c4aa0 2023-03-03T06:18:14.779 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35400c3610 con 0x7f35400c4aa0 2023-03-03T06:18:14.779 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.763+0000 7f35367fc700 1 -- 172.21.15.90:0/922094354 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0038c0 con 0x7f35400c4aa0 2023-03-03T06:18:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.764+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35400c3330 con 0x7f35400c4aa0 2023-03-03T06:18:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.765+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35401535c0 con 0x7f35400c4aa0 2023-03-03T06:18:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.765+0000 7f35367fc700 1 -- 172.21.15.90:0/922094354 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f353c019030 con 0x7f35400c4aa0 2023-03-03T06:18:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.765+0000 7f35367fc700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f35240682f0 0x7f352406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.765+0000 7f35367fc700 1 -- 172.21.15.90:0/922094354 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f353c085060 con 0x7f35400c4aa0 2023-03-03T06:18:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.765+0000 7f35451f1700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f35240682f0 0x7f352406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.766+0000 7f35451f1700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f35240682f0 0x7f352406a7a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35400a6420 tx=0x7f3530008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:14.782 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.769+0000 7f35367fc700 1 -- 172.21.15.90:0/922094354 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f353c051860 con 0x7f35400c4aa0 2023-03-03T06:18:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.994+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f354004e750 con 0x7f35400c4aa0 2023-03-03T06:18:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.995+0000 7f35367fc700 1 -- 172.21.15.90:0/922094354 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f353c054e80 con 0x7f35400c4aa0 2023-03-03T06:18:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:14.997 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-03-03T06:18:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f35240682f0 msgr2=0x7f352406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.999 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f35240682f0 0x7f352406a7a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35400a6420 tx=0x7f3530008040 comp rx=0 tx=0).stop 2023-03-03T06:18:14.999 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35400c4aa0 msgr2=0x7f35400bd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:14.999 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35400c4aa0 0x7f35400bd6f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c510 tx=0x7f353c00c8d0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.999 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 shutdown_connections 2023-03-03T06:18:14.999 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35401531f0 0x7f35401e7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:14.999 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f35240682f0 0x7f352406a7a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.000 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35400c4aa0 0x7f35400bd6f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.000 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 --2- 172.21.15.90:0/922094354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35401506b0 0x7f35400b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.000 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 >> 172.21.15.90:0/922094354 conn(0x7f35400a7060 msgr2=0x7f35400c0150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:15.000 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 shutdown_connections 2023-03-03T06:18:15.000 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:14.997+0000 7f3547c56700 1 -- 172.21.15.90:0/922094354 wait complete. 2023-03-03T06:18:15.010 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-03-03T06:18:15.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T06:18:15.129 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.127+0000 7f0eb3edb700 1 Processor -- start 2023-03-03T06:18:15.129 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.127+0000 7f0eb3edb700 1 -- start start 2023-03-03T06:18:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.128+0000 7f0eb3edb700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac14f7d0 0x7f0eac15aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.128+0000 7f0eb3edb700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac153340 con 0x7f0eac06ea90 2023-03-03T06:18:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.128+0000 7f0eb3edb700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac1497f0 con 0x7f0eac149420 2023-03-03T06:18:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.128+0000 7f0eb3edb700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac152750 con 0x7f0eac14f7d0 2023-03-03T06:18:15.131 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.128+0000 7f0eb1476700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0eac06ea90 0x7f0eac14f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53634/0 (socket says 172.21.15.90:53634) 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.128+0000 7f0eb1476700 1 -- 172.21.15.90:0/2482659962 learned_addr learned my addr 172.21.15.90:0/2482659962 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb2478700 1 --2- 172.21.15.90:0/2482659962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac14f7d0 0x7f0eac15aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902511492 0 0) 0x7f0eac153340 con 0x7f0eac06ea90 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e980035b0 con 0x7f0eac06ea90 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067739099 0 0) 0x7f0eac1497f0 con 0x7f0eac149420 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eac153340 con 0x7f0eac149420 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1018718720 0 0) 0x7f0e980035b0 con 0x7f0eac06ea90 2023-03-03T06:18:15.132 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb2478700 1 -- 172.21.15.90:0/2482659962 >> v1:172.21.15.121:6789/0 conn(0x7f0eac149420 legacy=0x7f0eac0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb2478700 1 -- 172.21.15.90:0/2482659962 >> v1:172.21.15.90:6789/0 conn(0x7f0eac06ea90 legacy=0x7f0eac14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb2478700 1 -- 172.21.15.90:0/2482659962 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac152a10 con 0x7f0eac14f7d0 2023-03-03T06:18:15.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb2478700 1 --2- 172.21.15.90:0/2482659962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac14f7d0 0x7f0eac15aa50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009dc0 tx=0x7f0ea400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e274ded2c3d1dac5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400e070 con 0x7f0eac14f7d0 2023-03-03T06:18:15.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea400bb20 con 0x7f0eac14f7d0 2023-03-03T06:18:15.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.129+0000 7f0eb0c75700 1 -- 172.21.15.90:0/2482659962 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400bed0 con 0x7f0eac14f7d0 2023-03-03T06:18:15.135 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.130+0000 7f0eb3edb700 1 -- 172.21.15.90:0/2482659962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac14f7d0 msgr2=0x7f0eac15aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.130+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/2482659962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac14f7d0 0x7f0eac15aa50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009dc0 tx=0x7f0ea400b330 comp rx=0 tx=0).stop 2023-03-03T06:18:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.130+0000 7f0eb3edb700 1 -- 172.21.15.90:0/2482659962 shutdown_connections 2023-03-03T06:18:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.130+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/2482659962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac14f7d0 0x7f0eac15aa50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.130+0000 7f0eb3edb700 1 -- 172.21.15.90:0/2482659962 >> 172.21.15.90:0/2482659962 conn(0x7f0eac06c0d0 msgr2=0x7f0eac0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.130+0000 7f0eb3edb700 1 -- 172.21.15.90:0/2482659962 shutdown_connections 2023-03-03T06:18:15.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.130+0000 7f0eb3edb700 1 -- 172.21.15.90:0/2482659962 wait complete. 2023-03-03T06:18:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.131+0000 7f0eb3edb700 1 Processor -- start 2023-03-03T06:18:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.131+0000 7f0eb3edb700 1 -- start start 2023-03-03T06:18:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb3edb700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac06ea90 0x7f0eac141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb3edb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac149420 0x7f0eac1e4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb3edb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eac14f7d0 0x7f0eac14e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb3edb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1e8f30 con 0x7f0eac149420 2023-03-03T06:18:15.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb3edb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1e5a40 con 0x7f0eac14f7d0 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb3edb700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1e9110 con 0x7f0eac06ea90 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb1476700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac149420 0x7f0eac1e4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb1476700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac149420 0x7f0eac1e4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34436/0 (socket says 172.21.15.90:34436) 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb1476700 1 -- 172.21.15.90:0/329937502 learned_addr learned my addr 172.21.15.90:0/329937502 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.132+0000 7f0eb1c77700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac06ea90 0x7f0eac141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb2478700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eac14f7d0 0x7f0eac14e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb1476700 1 -- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac06ea90 msgr2=0x7f0eac141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb1476700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac06ea90 0x7f0eac141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb1476700 1 -- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eac14f7d0 msgr2=0x7f0eac14e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb1476700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eac14f7d0 0x7f0eac14e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb1476700 1 -- 172.21.15.90:0/329937502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4009a70 con 0x7f0eac149420 2023-03-03T06:18:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb2478700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eac14f7d0 0x7f0eac14e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb1c77700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac06ea90 0x7f0eac141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb1476700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac149420 0x7f0eac1e4fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800c4b0 tx=0x7f0ea800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0ea27fc700 1 -- 172.21.15.90:0/329937502 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8005d70 con 0x7f0eac149420 2023-03-03T06:18:15.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0ea27fc700 1 -- 172.21.15.90:0/329937502 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea80070d0 con 0x7f0eac149420 2023-03-03T06:18:15.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac152fd0 con 0x7f0eac149420 2023-03-03T06:18:15.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0ea27fc700 1 -- 172.21.15.90:0/329937502 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8007460 con 0x7f0eac149420 2023-03-03T06:18:15.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.133+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac152cf0 con 0x7f0eac149420 2023-03-03T06:18:15.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.134+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac06e610 con 0x7f0eac149420 2023-03-03T06:18:15.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.134+0000 7f0ea27fc700 1 -- 172.21.15.90:0/329937502 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0ea8019030 con 0x7f0eac149420 2023-03-03T06:18:15.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.135+0000 7f0ea27fc700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0e900682f0 0x7f0e9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.135+0000 7f0eb1c77700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0e900682f0 0x7f0e9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.135+0000 7f0ea27fc700 1 -- 172.21.15.90:0/329937502 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0ea80851c0 con 0x7f0eac149420 2023-03-03T06:18:15.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.135+0000 7f0eb1c77700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0e900682f0 0x7f0e9006a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c007fd0 tx=0x7f0e9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.139+0000 7f0ea27fc700 1 -- 172.21.15.90:0/329937502 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ea80518b0 con 0x7f0eac149420 2023-03-03T06:18:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.327+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0eac04e750 con 0x7f0eac149420 2023-03-03T06:18:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.328+0000 7f0ea27fc700 1 -- 172.21.15.90:0/329937502 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17811 (secure 0 0 0) 0x7f0ea8054ed0 con 0x7f0eac149420 2023-03-03T06:18:15.330 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:15.330 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":23,"fsid":"05bb16be-ef0c-4f48-a742-b70017592324","created":"2023-03-03T06:17:40.554816+0000","modified":"2023-03-03T06:18:06.754461+0000","last_up_change":"2023-03-03T06:17:51.956492+0000","last_in_change":"2023-03-03T06:17:43.077431+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T06:17:52.113684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T06:17:58.969014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-03-03T06:18:03.400568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-03-03T06:18:04.718088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c28f7dbf-67e2-451d-804b-7c735332e79d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":96747},{"type":"v1","addr":"172.21.15.90:6815","nonce":96747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":96747},{"type":"v1","addr":"172.21.15.90:6819","nonce":96747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":96747},{"type":"v1","addr":"172.21.15.90:6823","nonce":96747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":96747},{"type":"v1","addr":"172.21.15.90:6821","nonce":96747}]},"public_addr":"172.21.15.90:6815/96747","cluster_addr":"172.21.15.90:6819/96747","heartbeat_back_addr":"172.21.15.90:6823/96747","heartbeat_front_addr":"172.21.15.90:6821/96747","state":["exists","up"]},{"osd":1,"uuid":"e685d0cd-4c2a-4d24-bf07-6bc9ecdec2b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":96746},{"type":"v1","addr":"172.21.15.90:6801","nonce":96746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":96746},{"type":"v1","addr":"172.21.15.90:6803","nonce":96746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":96746},{"type":"v1","addr":"172.21.15.90:6807","nonce":96746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":96746},{"type":"v1","addr":"172.21.15.90:6805","nonce":96746}]},"public_addr":"172.21.15.90:6801/96746","cluster_addr":"172.21.15.90:6803/96746","heartbeat_back_addr":"172.21.15.90:6807/96746","heartbeat_front_addr":"172.21.15.90:6805/96746","state":["exists","up"]},{"osd":2,"uuid":"c3f69062-6ead-4f52-b01b-6213a6adfdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":96756},{"type":"v1","addr":"172.21.15.90:6809","nonce":96756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":96756},{"type":"v1","addr":"172.21.15.90:6811","nonce":96756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":96756},{"type":"v1","addr":"172.21.15.90:6818","nonce":96756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":96756},{"type":"v1","addr":"172.21.15.90:6814","nonce":96756}]},"public_addr":"172.21.15.90:6809/96756","cluster_addr":"172.21.15.90:6811/96756","heartbeat_back_addr":"172.21.15.90:6818/96756","heartbeat_front_addr":"172.21.15.90:6814/96756","state":["exists","up"]},{"osd":3,"uuid":"80d84903-90c6-4a8d-aa31-6bcbe1c5c5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":96763},{"type":"v1","addr":"172.21.15.90:6825","nonce":96763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":96763},{"type":"v1","addr":"172.21.15.90:6827","nonce":96763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":96763},{"type":"v1","addr":"172.21.15.90:6831","nonce":96763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":96763},{"type":"v1","addr":"172.21.15.90:6829","nonce":96763}]},"public_addr":"172.21.15.90:6825/96763","cluster_addr":"172.21.15.90:6827/96763","heartbeat_back_addr":"172.21.15.90:6831/96763","heartbeat_front_addr":"172.21.15.90:6829/96763","state":["exists","up"]},{"osd":4,"uuid":"68014d03-1e69-4652-a555-5a524933f4a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":96843},{"type":"v1","addr":"172.21.15.121:6817","nonce":96843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":96843},{"type":"v1","addr":"172.21.15.121:6819","nonce":96843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":96843},{"type":"v1","addr":"172.21.15.121:6823","nonce":96843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":96843},{"type":"v1","addr":"172.21.15.121:6821","nonce":96843}]},"public_addr":"172.21.15.121:6817/96843","cluster_addr":"172.21.15.121:6819/96843","heartbeat_back_addr":"172.21.15.121:6823/96843","heartbeat_front_addr":"172.21.15.121:6821/96843","state":["exists","up"]},{"osd":5,"uuid":"4c94930d-c84f-4f40-a615-d42ab7d8323a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":96841},{"type":"v1","addr":"172.21.15.121:6801","nonce":96841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":96841},{"type":"v1","addr":"172.21.15.121:6803","nonce":96841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":96841},{"type":"v1","addr":"172.21.15.121:6807","nonce":96841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":96841},{"type":"v1","addr":"172.21.15.121:6805","nonce":96841}]},"public_addr":"172.21.15.121:6801/96841","cluster_addr":"172.21.15.121:6803/96841","heartbeat_back_addr":"172.21.15.121:6807/96841","heartbeat_front_addr":"172.21.15.121:6805/96841","state":["exists","up"]},{"osd":6,"uuid":"3c0c3008-2ed7-4db3-9b17-3226864daff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":96842},{"type":"v1","addr":"172.21.15.121:6809","nonce":96842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":96842},{"type":"v1","addr":"172.21.15.121:6811","nonce":96842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":96842},{"type":"v1","addr":"172.21.15.121:6815","nonce":96842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":96842},{"type":"v1","addr":"172.21.15.121:6813","nonce":96842}]},"public_addr":"172.21.15.121:6809/96842","cluster_addr":"172.21.15.121:6811/96842","heartbeat_back_addr":"172.21.15.121:6815/96842","heartbeat_front_addr":"172.21.15.121:6813/96842","state":["exists","up"]},{"osd":7,"uuid":"aa96b041-9dcd-4888-9b17-4cdaaa907e4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":96852},{"type":"v1","addr":"172.21.15.121:6825","nonce":96852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":96852},{"type":"v1","addr":"172.21.15.121:6827","nonce":96852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":96852},{"type":"v1","addr":"172.21.15.121:6831","nonce":96852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":96852},{"type":"v1","addr":"172.21.15.121:6829","nonce":96852}]},"public_addr":"172.21.15.121:6825/96852","cluster_addr":"172.21.15.121:6827/96852","heartbeat_back_addr":"172.21.15.121:6831/96852","heartbeat_front_addr":"172.21.15.121:6829/96852","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.336107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.260471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.257793+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.289652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.274258+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.208985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.213919+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T06:17:48.215502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T06:18:15.332 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0e900682f0 msgr2=0x7f0e9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.332 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0e900682f0 0x7f0e9006a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c007fd0 tx=0x7f0e9c009040 comp rx=0 tx=0).stop 2023-03-03T06:18:15.332 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac149420 msgr2=0x7f0eac1e4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.332 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac149420 0x7f0eac1e4fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800c4b0 tx=0x7f0ea800c870 comp rx=0 tx=0).stop 2023-03-03T06:18:15.332 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 shutdown_connections 2023-03-03T06:18:15.332 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0eac14f7d0 0x7f0eac14e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0eac06ea90 0x7f0eac141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f0e900682f0 0x7f0e9006a7a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 --2- 172.21.15.90:0/329937502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac149420 0x7f0eac1e4fb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 >> 172.21.15.90:0/329937502 conn(0x7f0eac06c0d0 msgr2=0x7f0eac0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:15.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 shutdown_connections 2023-03-03T06:18:15.333 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.330+0000 7f0eb3edb700 1 -- 172.21.15.90:0/329937502 wait complete. 2023-03-03T06:18:15.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T06:17:52.113684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T06:17:58.969014+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-03T06:18:03.400568+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-03-03T06:18:04.718088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T06:18:15.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T06:18:15.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.466+0000 7feea0f0c700 1 Processor -- start 2023-03-03T06:18:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.466+0000 7feea0f0c700 1 -- start start 2023-03-03T06:18:15.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7feea0f0c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee9c149420 0x7fee9c0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7feea0f0c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c06ea90 0x7fee9c159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7feea0f0c700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c153340 con 0x7fee9c0c2da0 2023-03-03T06:18:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7feea0f0c700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c1497f0 con 0x7fee9c149420 2023-03-03T06:18:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7feea0f0c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c152750 con 0x7fee9c06ea90 2023-03-03T06:18:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee9c149420 0x7fee9c0c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee9c149420 0x7fee9c0c2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34442/0 (socket says 172.21.15.90:34442) 2023-03-03T06:18:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 -- 172.21.15.90:0/3670606141 learned_addr learned my addr 172.21.15.90:0/3670606141 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9ad9d700 1 --2- 172.21.15.90:0/3670606141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c06ea90 0x7fee9c159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 -- 172.21.15.90:0/3670606141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c06ea90 msgr2=0x7fee9c159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 --2- 172.21.15.90:0/3670606141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c06ea90 0x7fee9c159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 -- 172.21.15.90:0/3670606141 >> v1:172.21.15.121:6789/0 conn(0x7fee9c0c2da0 legacy=0x7fee9c14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 -- 172.21.15.90:0/3670606141 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c152a10 con 0x7fee9c149420 2023-03-03T06:18:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.467+0000 7fee9a59c700 1 --2- 172.21.15.90:0/3670606141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c0c2860 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fee900098f0 tx=0x7fee90009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826f52b62b835bf2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7fee9959a700 1 -- 172.21.15.90:0/3670606141 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9000b8a0 con 0x7fee9c149420 2023-03-03T06:18:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7fee9959a700 1 -- 172.21.15.90:0/3670606141 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9000ba00 con 0x7fee9c149420 2023-03-03T06:18:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7fee9959a700 1 -- 172.21.15.90:0/3670606141 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9000bd90 con 0x7fee9c149420 2023-03-03T06:18:15.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 -- 172.21.15.90:0/3670606141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 msgr2=0x7fee9c0c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3670606141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c0c2860 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fee900098f0 tx=0x7fee90009c00 comp rx=0 tx=0).stop 2023-03-03T06:18:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 -- 172.21.15.90:0/3670606141 shutdown_connections 2023-03-03T06:18:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3670606141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c06ea90 0x7fee9c159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3670606141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c0c2860 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 -- 172.21.15.90:0/3670606141 >> 172.21.15.90:0/3670606141 conn(0x7fee9c06c0d0 msgr2=0x7fee9c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 -- 172.21.15.90:0/3670606141 shutdown_connections 2023-03-03T06:18:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.468+0000 7feea0f0c700 1 -- 172.21.15.90:0/3670606141 wait complete. 2023-03-03T06:18:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.469+0000 7feea0f0c700 1 Processor -- start 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7feea0f0c700 1 -- start start 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7feea0f0c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee9c06ea90 0x7fee9c146a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7feea0f0c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c0c2da0 0x7fee9c1de540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7feea0f0c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c1dea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7feea0f0c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c1514c0 con 0x7fee9c149420 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7feea0f0c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c149ae0 con 0x7fee9c06ea90 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7feea0f0c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0a6720 con 0x7fee9c0c2da0 2023-03-03T06:18:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7fee9ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c1dea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7fee9ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c1dea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34450/0 (socket says 172.21.15.90:34450) 2023-03-03T06:18:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7fee9ad9d700 1 -- 172.21.15.90:0/3801929913 learned_addr learned my addr 172.21.15.90:0/3801929913 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7fee99d9b700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c0c2da0 0x7fee9c1de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.470+0000 7fee9a59c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee9c06ea90 0x7fee9c146a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee9ad9d700 1 -- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c0c2da0 msgr2=0x7fee9c1de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee9ad9d700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c0c2da0 0x7fee9c1de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee9ad9d700 1 -- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee9c06ea90 msgr2=0x7fee9c146a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee9ad9d700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee9c06ea90 0x7fee9c146a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee9ad9d700 1 -- 172.21.15.90:0/3801929913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee90009550 con 0x7fee9c149420 2023-03-03T06:18:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee9ad9d700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c1dea80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fee94007e90 tx=0x7fee9400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee8affd700 1 -- 172.21.15.90:0/3801929913 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9400ed30 con 0x7fee9c149420 2023-03-03T06:18:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee8affd700 1 -- 172.21.15.90:0/3801929913 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9400ee90 con 0x7fee9c149420 2023-03-03T06:18:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c152fd0 con 0x7fee9c149420 2023-03-03T06:18:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7fee8affd700 1 -- 172.21.15.90:0/3801929913 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94014660 con 0x7fee9c149420 2023-03-03T06:18:15.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.471+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c152cf0 con 0x7fee9c149420 2023-03-03T06:18:15.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.472+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c06e610 con 0x7fee9c149420 2023-03-03T06:18:15.478 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.472+0000 7fee8affd700 1 -- 172.21.15.90:0/3801929913 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fee94012050 con 0x7fee9c149420 2023-03-03T06:18:15.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.472+0000 7fee8affd700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fee7c0682f0 0x7fee7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.473+0000 7fee8affd700 1 -- 172.21.15.90:0/3801929913 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fee94086100 con 0x7fee9c149420 2023-03-03T06:18:15.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.473+0000 7fee9a59c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fee7c0682f0 0x7fee7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.473+0000 7fee9a59c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fee7c0682f0 0x7fee7c06a7a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fee90009c40 tx=0x7fee90008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.479 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.477+0000 7fee8affd700 1 -- 172.21.15.90:0/3801929913 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fee94052900 con 0x7fee9c149420 2023-03-03T06:18:15.663 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.661+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee9c149c50 con 0x7fee9c149420 2023-03-03T06:18:15.664 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.662+0000 7fee8affd700 1 -- 172.21.15.90:0/3801929913 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee94055f20 con 0x7fee9c149420 2023-03-03T06:18:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2023-03-03T06:18:15.666 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fee7c0682f0 msgr2=0x7fee7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.666 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fee7c0682f0 0x7fee7c06a7a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fee90009c40 tx=0x7fee90008030 comp rx=0 tx=0).stop 2023-03-03T06:18:15.666 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 msgr2=0x7fee9c1dea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.666 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c1dea80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fee94007e90 tx=0x7fee9400e3c0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.666 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 shutdown_connections 2023-03-03T06:18:15.667 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fee7c0682f0 0x7fee7c06a7a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.667 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee9c149420 0x7fee9c1dea80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.667 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee9c06ea90 0x7fee9c146a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.667 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 --2- 172.21.15.90:0/3801929913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee9c0c2da0 0x7fee9c1de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.667 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 >> 172.21.15.90:0/3801929913 conn(0x7fee9c06c0d0 msgr2=0x7fee9c0bef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:15.667 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 shutdown_connections 2023-03-03T06:18:15.667 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.664+0000 7feea0f0c700 1 -- 172.21.15.90:0/3801929913 wait complete. 2023-03-03T06:18:15.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T06:18:15.807 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.799+0000 7f02b2704700 1 Processor -- start 2023-03-03T06:18:15.808 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.799+0000 7f02b2704700 1 -- start start 2023-03-03T06:18:15.808 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02b2704700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 0x7f02ac14f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.808 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02b2704700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac153340 con 0x7f02ac149420 2023-03-03T06:18:15.808 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02b2704700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac1497f0 con 0x7f02ac14f600 2023-03-03T06:18:15.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02b2704700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac152750 con 0x7f02ac06ea90 2023-03-03T06:18:15.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02abfff700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f02ac149420 0x7f02ac0c2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53660/0 (socket says 172.21.15.90:53660) 2023-03-03T06:18:15.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02abfff700 1 -- 172.21.15.90:0/522957892 learned_addr learned my addr 172.21.15.90:0/522957892 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:15.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02ab7fe700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 0x7f02ac14f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951398870 0 0) 0x7f02ac153340 con 0x7f02ac149420 2023-03-03T06:18:15.809 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02940035b0 con 0x7f02ac149420 2023-03-03T06:18:15.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784599696 0 0) 0x7f02940035b0 con 0x7f02ac149420 2023-03-03T06:18:15.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 msgr2=0x7f02ac14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:15.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 0x7f02ac14f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 >> v1:172.21.15.121:6789/0 conn(0x7f02ac14f600 legacy=0x7f02ac159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac152a10 con 0x7f02ac149420 2023-03-03T06:18:15.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 337907801 0 0) 0x7f02a0001670 con 0x7f02ac149420 2023-03-03T06:18:15.810 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 >> v1:172.21.15.90:6789/0 conn(0x7f02ac149420 legacy=0x7f02ac0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0294003a30 0x7f0294003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 0x7f0294004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0294004d90 0x7f0294005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac152750 con 0x7f0294004d90 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029400bb20 con 0x7f0294003a30 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.800+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029400bce0 con 0x7f02ac06ea90 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02b0ca1700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0294004d90 0x7f0294005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.811 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02abfff700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0294003a30 0x7f0294003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.812 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02ab7fe700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 0x7f0294004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.813 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02b0ca1700 1 -- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 msgr2=0x7f0294004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:15.813 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02b0ca1700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac06ea90 0x7f0294004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.813 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02b0ca1700 1 -- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0294003a30 msgr2=0x7f0294003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:15.813 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02b0ca1700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0294003a30 0x7f0294003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.814 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02b0ca1700 1 -- 172.21.15.90:0/522957892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02940035b0 con 0x7f0294004d90 2023-03-03T06:18:15.814 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02b0ca1700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0294004d90 0x7f0294005200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c240 tx=0x7f0294005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3e7a14ec8868b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.814 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4001e70 con 0x7f0294004d90 2023-03-03T06:18:15.814 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a400cde0 con 0x7f0294004d90 2023-03-03T06:18:15.814 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.801+0000 7f02aaffd700 1 -- 172.21.15.90:0/522957892 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4012660 con 0x7f0294004d90 2023-03-03T06:18:15.814 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.802+0000 7f02b2704700 1 -- 172.21.15.90:0/522957892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0294004d90 msgr2=0x7f0294005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.814 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.802+0000 7f02b2704700 1 --2- 172.21.15.90:0/522957892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0294004d90 0x7f0294005200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c240 tx=0x7f0294005880 comp rx=0 tx=0).stop 2023-03-03T06:18:15.815 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.802+0000 7f02b0ca1700 1 -- 172.21.15.90:0/522957892 reap_dead start 2023-03-03T06:18:15.815 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.802+0000 7f02b2704700 1 -- 172.21.15.90:0/522957892 shutdown_connections 2023-03-03T06:18:15.815 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.802+0000 7f02b2704700 1 -- 172.21.15.90:0/522957892 >> 172.21.15.90:0/522957892 conn(0x7f02ac06c0d0 msgr2=0x7f02ac0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:15.815 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.802+0000 7f02b2704700 1 -- 172.21.15.90:0/522957892 shutdown_connections 2023-03-03T06:18:15.815 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.802+0000 7f02b2704700 1 -- 172.21.15.90:0/522957892 wait complete. 2023-03-03T06:18:15.815 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.803+0000 7f02b2704700 1 Processor -- start 2023-03-03T06:18:15.815 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.803+0000 7f02b2704700 1 -- start start 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.803+0000 7f02b2704700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac1e8210 0x7f02ac1e85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.803+0000 7f02b2704700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02ac0a78c0 0x7f02ac1480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b2704700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02ac148630 0x7f02ac1e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b2704700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1e8e70 con 0x7f02ac148630 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b2704700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1e8b20 con 0x7f02ac0a78c0 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b2704700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac14c6f0 con 0x7f02ac1e8210 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02ac148630 0x7f02ac1e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.816 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02ac148630 0x7f02ac1e4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34470/0 (socket says 172.21.15.90:34470) 2023-03-03T06:18:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 -- 172.21.15.90:0/1409235045 learned_addr learned my addr 172.21.15.90:0/1409235045 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02abfff700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac1e8210 0x7f02ac1e85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02ab7fe700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02ac0a78c0 0x7f02ac1480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 -- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac1e8210 msgr2=0x7f02ac1e85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac1e8210 0x7f02ac1e85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 -- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02ac0a78c0 msgr2=0x7f02ac1480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02ac0a78c0 0x7f02ac1480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 -- 172.21.15.90:0/1409235045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac152fd0 con 0x7f02ac148630 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b0ca1700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02ac148630 0x7f02ac1e4cf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f029400baf0 tx=0x7f02a4001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02a8ff9700 1 -- 172.21.15.90:0/1409235045 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a40012d0 con 0x7f02ac148630 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02a8ff9700 1 -- 172.21.15.90:0/1409235045 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4020070 con 0x7f02ac148630 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac152cf0 con 0x7f02ac148630 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02a8ff9700 1 -- 172.21.15.90:0/1409235045 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a40050c0 con 0x7f02ac148630 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.804+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac06e610 con 0x7f02ac148630 2023-03-03T06:18:15.818 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.805+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac1425f0 con 0x7f02ac148630 2023-03-03T06:18:15.819 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.806+0000 7f02a8ff9700 1 -- 172.21.15.90:0/1409235045 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f02a402b020 con 0x7f02ac148630 2023-03-03T06:18:15.819 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.806+0000 7f02a8ff9700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f02980682f0 0x7f029806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:15.819 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.806+0000 7f02a8ff9700 1 -- 172.21.15.90:0/1409235045 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f02a4086e90 con 0x7f02ac148630 2023-03-03T06:18:15.819 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.806+0000 7f02abfff700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f02980682f0 0x7f029806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:15.820 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.806+0000 7f02abfff700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f02980682f0 0x7f029806a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02a0000c00 tx=0x7f02a0001c40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:15.820 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.810+0000 7f02a8ff9700 1 -- 172.21.15.90:0/1409235045 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f02ac1425f0 con 0x7f02ac148630 2023-03-03T06:18:15.993 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.991+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f02ac1425f0 con 0x7f02ac148630 2023-03-03T06:18:15.994 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.992+0000 7f02a8ff9700 1 -- 172.21.15.90:0/1409235045 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f02ac1425f0 con 0x7f02ac148630 2023-03-03T06:18:15.994 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 8 2023-03-03T06:18:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f02980682f0 msgr2=0x7f029806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f02980682f0 0x7f029806a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02a0000c00 tx=0x7f02a0001c40 comp rx=0 tx=0).stop 2023-03-03T06:18:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02ac148630 msgr2=0x7f02ac1e4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02ac148630 0x7f02ac1e4cf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f029400baf0 tx=0x7f02a4001490 comp rx=0 tx=0).stop 2023-03-03T06:18:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 shutdown_connections 2023-03-03T06:18:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f02980682f0 0x7f029806a7a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02ac148630 0x7f02ac1e4cf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac1e8210 0x7f02ac1e85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 --2- 172.21.15.90:0/1409235045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02ac0a78c0 0x7f02ac1480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 >> 172.21.15.90:0/1409235045 conn(0x7f02ac06c0d0 msgr2=0x7f02ac0beb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 shutdown_connections 2023-03-03T06:18:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:15.994+0000 7f02b2704700 1 -- 172.21.15.90:0/1409235045 wait complete. 2023-03-03T06:18:16.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-03T06:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.128+0000 7f07783d3700 1 Processor -- start 2023-03-03T06:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.128+0000 7f07783d3700 1 -- start start 2023-03-03T06:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f07783d3700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0770145700 0x7f077006daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f07783d3700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077014b4e0 con 0x7f07700c11d0 2023-03-03T06:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f07783d3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07700a0ed0 con 0x7f077014bcc0 2023-03-03T06:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f07783d3700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07700bd330 con 0x7f0770145700 2023-03-03T06:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f077616f700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0770145700 0x7f077006daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f077616f700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0770145700 0x7f077006daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34474/0 (socket says 172.21.15.90:34474) 2023-03-03T06:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f077616f700 1 -- 172.21.15.90:0/2391154788 learned_addr learned my addr 172.21.15.90:0/2391154788 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.129+0000 7f077596e700 1 --1- 172.21.15.90:0/2391154788 >> v1:172.21.15.90:6789/0 conn(0x7f07700c11d0 0x7f077006cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53680/0 (socket says 172.21.15.90:53680) 2023-03-03T06:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f077616f700 1 -- 172.21.15.90:0/2391154788 >> v1:172.21.15.121:6789/0 conn(0x7f077014bcc0 legacy=0x7f077014ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f077616f700 1 -- 172.21.15.90:0/2391154788 >> v1:172.21.15.90:6789/0 conn(0x7f07700c11d0 legacy=0x7f077006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f077616f700 1 -- 172.21.15.90:0/2391154788 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07700a11b0 con 0x7f0770145700 2023-03-03T06:18:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f077616f700 1 --2- 172.21.15.90:0/2391154788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0770145700 0x7f077006daa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f076000bb90 tx=0x7f076000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a101d3587c3920 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f077516d700 1 -- 172.21.15.90:0/2391154788 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076000dc40 con 0x7f0770145700 2023-03-03T06:18:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f077516d700 1 -- 172.21.15.90:0/2391154788 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076000dda0 con 0x7f0770145700 2023-03-03T06:18:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f077516d700 1 -- 172.21.15.90:0/2391154788 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0760017630 con 0x7f0770145700 2023-03-03T06:18:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f07783d3700 1 -- 172.21.15.90:0/2391154788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0770145700 msgr2=0x7f077006daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f07783d3700 1 --2- 172.21.15.90:0/2391154788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0770145700 0x7f077006daa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f076000bb90 tx=0x7f076000bea0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f07783d3700 1 -- 172.21.15.90:0/2391154788 shutdown_connections 2023-03-03T06:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f07783d3700 1 --2- 172.21.15.90:0/2391154788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0770145700 0x7f077006daa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f07783d3700 1 -- 172.21.15.90:0/2391154788 >> 172.21.15.90:0/2391154788 conn(0x7f07700c1900 msgr2=0x7f07700c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.130+0000 7f07783d3700 1 -- 172.21.15.90:0/2391154788 shutdown_connections 2023-03-03T06:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.131+0000 7f07783d3700 1 -- 172.21.15.90:0/2391154788 wait complete. 2023-03-03T06:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 Processor -- start 2023-03-03T06:18:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 -- start start 2023-03-03T06:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07700c11d0 0x7f077014a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0770145700 0x7f07700b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f077014bcc0 0x7f07700b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07701d3ba0 con 0x7f077014bcc0 2023-03-03T06:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07700b85a0 con 0x7f07700c11d0 2023-03-03T06:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f07783d3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07701d5440 con 0x7f0770145700 2023-03-03T06:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f0776970700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f077014bcc0 0x7f07700b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f0776970700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f077014bcc0 0x7f07700b74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34484/0 (socket says 172.21.15.90:34484) 2023-03-03T06:18:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.132+0000 7f0776970700 1 -- 172.21.15.90:0/3111452880 learned_addr learned my addr 172.21.15.90:0/3111452880 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f077596e700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0770145700 0x7f07700b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f077616f700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07700c11d0 0x7f077014a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0776970700 1 -- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0770145700 msgr2=0x7f07700b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0776970700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0770145700 0x7f07700b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0776970700 1 -- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07700c11d0 msgr2=0x7f077014a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0776970700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07700c11d0 0x7f077014a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0776970700 1 -- 172.21.15.90:0/3111452880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076000b7f0 con 0x7f077014bcc0 2023-03-03T06:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f077596e700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0770145700 0x7f07700b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0776970700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f077014bcc0 0x7f07700b74a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f076800cc50 tx=0x7f076800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0766ffd700 1 -- 172.21.15.90:0/3111452880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768003c00 con 0x7f077014bcc0 2023-03-03T06:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0766ffd700 1 -- 172.21.15.90:0/3111452880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0768003d60 con 0x7f077014bcc0 2023-03-03T06:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077006d220 con 0x7f077014bcc0 2023-03-03T06:18:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f0766ffd700 1 -- 172.21.15.90:0/3111452880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076800ab60 con 0x7f077014bcc0 2023-03-03T06:18:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.133+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077014b7a0 con 0x7f077014bcc0 2023-03-03T06:18:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.134+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0770145ad0 con 0x7f077014bcc0 2023-03-03T06:18:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.134+0000 7f0766ffd700 1 -- 172.21.15.90:0/3111452880 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0768010050 con 0x7f077014bcc0 2023-03-03T06:18:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.135+0000 7f0766ffd700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f07540682f0 0x7f075406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.135+0000 7f077616f700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f07540682f0 0x7f075406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.135+0000 7f0766ffd700 1 -- 172.21.15.90:0/3111452880 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0768085b50 con 0x7f077014bcc0 2023-03-03T06:18:16.143 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.135+0000 7f077616f700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f07540682f0 0x7f075406a7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0760019040 tx=0x7f0760014000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.143 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.139+0000 7f0766ffd700 1 -- 172.21.15.90:0/3111452880 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07680523b0 con 0x7f077014bcc0 2023-03-03T06:18:16.326 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.324+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f077014aa10 con 0x7f077014bcc0 2023-03-03T06:18:16.326 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.325+0000 7f0766ffd700 1 -- 172.21.15.90:0/3111452880 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07680559d0 con 0x7f077014bcc0 2023-03-03T06:18:16.327 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2023-03-03T06:18:16.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f07540682f0 msgr2=0x7f075406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f07540682f0 0x7f075406a7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0760019040 tx=0x7f0760014000 comp rx=0 tx=0).stop 2023-03-03T06:18:16.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f077014bcc0 msgr2=0x7f07700b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f077014bcc0 0x7f07700b74a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f076800cc50 tx=0x7f076800cf60 comp rx=0 tx=0).stop 2023-03-03T06:18:16.329 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 shutdown_connections 2023-03-03T06:18:16.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f07540682f0 0x7f075406a7a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.330 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f077014bcc0 0x7f07700b74a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07700c11d0 0x7f077014a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 --2- 172.21.15.90:0/3111452880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0770145700 0x7f07700b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 >> 172.21.15.90:0/3111452880 conn(0x7f07700c1900 msgr2=0x7f07700bfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:16.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 shutdown_connections 2023-03-03T06:18:16.331 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.327+0000 7f07783d3700 1 -- 172.21.15.90:0/3111452880 wait complete. 2023-03-03T06:18:16.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-03T06:18:16.463 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 Processor -- start 2023-03-03T06:18:16.463 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 -- start start 2023-03-03T06:18:16.465 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff22c149420 0x7ff22c0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.465 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c14f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.465 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff22c06ea90 0x7ff22c14f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22c14fb40 con 0x7ff22c06ea90 2023-03-03T06:18:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22c153340 con 0x7ff22c149420 2023-03-03T06:18:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff232175700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c1497f0 con 0x7ff22c0c2da0 2023-03-03T06:18:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff22bfff700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff22c06ea90 0x7ff22c14f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff22bfff700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff22c06ea90 0x7ff22c14f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34492/0 (socket says 172.21.15.90:34492) 2023-03-03T06:18:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.462+0000 7ff22bfff700 1 -- 172.21.15.90:0/1542962411 learned_addr learned my addr 172.21.15.90:0/1542962411 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:16.466 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22affd700 1 --2- 172.21.15.90:0/1542962411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c14f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22b7fe700 1 --2- 172.21.15.90:0/1542962411 >> v2:172.21.15.121:3300/0 conn(0x7ff22c149420 0x7ff22c0c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22bfff700 1 -- 172.21.15.90:0/1542962411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 msgr2=0x7ff22c14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22bfff700 1 --2- 172.21.15.90:0/1542962411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c14f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22bfff700 1 -- 172.21.15.90:0/1542962411 >> v2:172.21.15.121:3300/0 conn(0x7ff22c149420 msgr2=0x7ff22c0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22bfff700 1 --2- 172.21.15.90:0/1542962411 >> v2:172.21.15.121:3300/0 conn(0x7ff22c149420 0x7ff22c0c2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22bfff700 1 -- 172.21.15.90:0/1542962411 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c152750 con 0x7ff22c06ea90 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22bfff700 1 --2- 172.21.15.90:0/1542962411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c06ea90 0x7ff22c14f600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff22c067e40 tx=0x7ff22400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3527b3f2ae901711 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.467 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22a7fc700 1 -- 172.21.15.90:0/1542962411 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22400e070 con 0x7ff22c06ea90 2023-03-03T06:18:16.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22a7fc700 1 -- 172.21.15.90:0/1542962411 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22400b9c0 con 0x7ff22c06ea90 2023-03-03T06:18:16.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff22a7fc700 1 -- 172.21.15.90:0/1542962411 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22400bd70 con 0x7ff22c06ea90 2023-03-03T06:18:16.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff232175700 1 -- 172.21.15.90:0/1542962411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c06ea90 msgr2=0x7ff22c14f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.463+0000 7ff232175700 1 --2- 172.21.15.90:0/1542962411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c06ea90 0x7ff22c14f600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff22c067e40 tx=0x7ff22400b330 comp rx=0 tx=0).stop 2023-03-03T06:18:16.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.464+0000 7ff232175700 1 -- 172.21.15.90:0/1542962411 shutdown_connections 2023-03-03T06:18:16.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.464+0000 7ff232175700 1 --2- 172.21.15.90:0/1542962411 >> v2:172.21.15.121:3300/0 conn(0x7ff22c149420 0x7ff22c0c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.468 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.464+0000 7ff232175700 1 --2- 172.21.15.90:0/1542962411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c06ea90 0x7ff22c14f600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.464+0000 7ff232175700 1 --2- 172.21.15.90:0/1542962411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c14f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.464+0000 7ff232175700 1 -- 172.21.15.90:0/1542962411 >> 172.21.15.90:0/1542962411 conn(0x7ff22c06c0d0 msgr2=0x7ff22c0c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:16.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.464+0000 7ff232175700 1 -- 172.21.15.90:0/1542962411 shutdown_connections 2023-03-03T06:18:16.469 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.464+0000 7ff232175700 1 -- 172.21.15.90:0/1542962411 wait complete. 2023-03-03T06:18:16.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.465+0000 7ff232175700 1 Processor -- start 2023-03-03T06:18:16.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.465+0000 7ff232175700 1 -- start start 2023-03-03T06:18:16.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff22c06ea90 0x7ff22c14bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c1e81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.471 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c149420 0x7ff22c1e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c1e7d90 con 0x7ff22c149420 2023-03-03T06:18:16.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c14ab20 con 0x7ff22c06ea90 2023-03-03T06:18:16.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c14c890 con 0x7ff22c0c2da0 2023-03-03T06:18:16.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c149420 0x7ff22c1e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c149420 0x7ff22c1e86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34498/0 (socket says 172.21.15.90:34498) 2023-03-03T06:18:16.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 -- 172.21.15.90:0/2405194077 learned_addr learned my addr 172.21.15.90:0/2405194077 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:16.472 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff22c06ea90 0x7ff22c14bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22affd700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c1e81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 -- 172.21.15.90:0/2405194077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 msgr2=0x7ff22c1e81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c1e81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 -- 172.21.15.90:0/2405194077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff22c06ea90 msgr2=0x7ff22c14bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff22c06ea90 0x7ff22c14bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 -- 172.21.15.90:0/2405194077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff224009b30 con 0x7ff22c149420 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff22bfff700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c149420 0x7ff22c1e86f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b370 tx=0x7ff224004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.473 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff223fff700 1 -- 172.21.15.90:0/2405194077 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224004250 con 0x7ff22c149420 2023-03-03T06:18:16.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff223fff700 1 -- 172.21.15.90:0/2405194077 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2240043f0 con 0x7ff22c149420 2023-03-03T06:18:16.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22c152cf0 con 0x7ff22c149420 2023-03-03T06:18:16.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff223fff700 1 -- 172.21.15.90:0/2405194077 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2240047a0 con 0x7ff22c149420 2023-03-03T06:18:16.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.466+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22c152a10 con 0x7ff22c149420 2023-03-03T06:18:16.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.467+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22c152fd0 con 0x7ff22c149420 2023-03-03T06:18:16.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.468+0000 7ff223fff700 1 -- 172.21.15.90:0/2405194077 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff22401b430 con 0x7ff22c149420 2023-03-03T06:18:16.474 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.468+0000 7ff223fff700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ff20c0682f0 0x7ff20c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.468+0000 7ff22b7fe700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ff20c0682f0 0x7ff20c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.468+0000 7ff223fff700 1 -- 172.21.15.90:0/2405194077 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff224086f80 con 0x7ff22c149420 2023-03-03T06:18:16.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.468+0000 7ff22b7fe700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ff20c0682f0 0x7ff20c06a7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00aae0 tx=0x7ff21c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.475 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.472+0000 7ff223fff700 1 -- 172.21.15.90:0/2405194077 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff224053780 con 0x7ff22c149420 2023-03-03T06:18:16.656 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.654+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff22c149ab0 con 0x7ff22c149420 2023-03-03T06:18:16.656 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.654+0000 7ff223fff700 1 -- 172.21.15.90:0/2405194077 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff22401f0a0 con 0x7ff22c149420 2023-03-03T06:18:16.657 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2023-03-03T06:18:16.658 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ff20c0682f0 msgr2=0x7ff20c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.658 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ff20c0682f0 0x7ff20c06a7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00aae0 tx=0x7ff21c008040 comp rx=0 tx=0).stop 2023-03-03T06:18:16.658 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c149420 msgr2=0x7ff22c1e86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.658 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c149420 0x7ff22c1e86f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b370 tx=0x7ff224004ad0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 shutdown_connections 2023-03-03T06:18:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7ff20c0682f0 0x7ff20c06a7a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22c149420 0x7ff22c1e86f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff22c06ea90 0x7ff22c14bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 --2- 172.21.15.90:0/2405194077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff22c0c2da0 0x7ff22c1e81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 >> 172.21.15.90:0/2405194077 conn(0x7ff22c06c0d0 msgr2=0x7ff22c0b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 shutdown_connections 2023-03-03T06:18:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.656+0000 7ff232175700 1 -- 172.21.15.90:0/2405194077 wait complete. 2023-03-03T06:18:16.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T06:18:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.788+0000 7f2f2b2a8700 1 Processor -- start 2023-03-03T06:18:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.788+0000 7f2f2b2a8700 1 -- start start 2023-03-03T06:18:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f2b2a8700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2f24151230 0x7f2f240bd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f2b2a8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f241559f0 0x7f2f24157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f2b2a8700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f240a0d30 con 0x7f2f240c0a20 2023-03-03T06:18:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f2b2a8700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f240c14e0 con 0x7f2f24151230 2023-03-03T06:18:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f2b2a8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240bfa60 con 0x7f2f241559f0 2023-03-03T06:18:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29044700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2f24151230 0x7f2f240bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29044700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2f24151230 0x7f2f240bd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34502/0 (socket says 172.21.15.90:34502) 2023-03-03T06:18:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29044700 1 -- 172.21.15.90:0/4262622561 learned_addr learned my addr 172.21.15.90:0/4262622561 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:16.794 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29845700 1 --2- 172.21.15.90:0/4262622561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f241559f0 0x7f2f24157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29044700 1 -- 172.21.15.90:0/4262622561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f241559f0 msgr2=0x7f2f24157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29044700 1 --2- 172.21.15.90:0/4262622561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f241559f0 0x7f2f24157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29044700 1 -- 172.21.15.90:0/4262622561 >> v1:172.21.15.121:6789/0 conn(0x7f2f240c0a20 legacy=0x7f2f2406cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.789+0000 7f2f29044700 1 -- 172.21.15.90:0/4262622561 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f24151600 con 0x7f2f24151230 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f29044700 1 --2- 172.21.15.90:0/4262622561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f24151230 0x7f2f240bd770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2f140098f0 tx=0x7f2f14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965c1ddd99eb604d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f1bfff700 1 -- 172.21.15.90:0/4262622561 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f14004230 con 0x7f2f24151230 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f1bfff700 1 -- 172.21.15.90:0/4262622561 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f14004390 con 0x7f2f24151230 2023-03-03T06:18:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f1bfff700 1 -- 172.21.15.90:0/4262622561 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f140036a0 con 0x7f2f24151230 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/4262622561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f24151230 msgr2=0x7f2f240bd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/4262622561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f24151230 0x7f2f240bd770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2f140098f0 tx=0x7f2f14009c00 comp rx=0 tx=0).stop 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/4262622561 shutdown_connections 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/4262622561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f241559f0 0x7f2f24157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/4262622561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f24151230 0x7f2f240bd770 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/4262622561 >> 172.21.15.90:0/4262622561 conn(0x7f2f240c1900 msgr2=0x7f2f240c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/4262622561 shutdown_connections 2023-03-03T06:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.790+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/4262622561 wait complete. 2023-03-03T06:18:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.791+0000 7f2f2b2a8700 1 Processor -- start 2023-03-03T06:18:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.791+0000 7f2f2b2a8700 1 -- start start 2023-03-03T06:18:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f2b2a8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f240c0a20 0x7f2f24143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f2b2a8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f24151230 0x7f2f240b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f2b2a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f241559f0 0x7f2f24141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f2b2a8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f241e4b70 con 0x7f2f240c0a20 2023-03-03T06:18:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f2b2a8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240b6de0 con 0x7f2f241559f0 2023-03-03T06:18:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f2b2a8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f24144330 con 0x7f2f24151230 2023-03-03T06:18:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f240c0a20 0x7f2f24143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f240c0a20 0x7f2f24143df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34506/0 (socket says 172.21.15.90:34506) 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 -- 172.21.15.90:0/3634869221 learned_addr learned my addr 172.21.15.90:0/3634869221 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 -- 172.21.15.90:0/3634869221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f24151230 msgr2=0x7f2f240b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f24151230 0x7f2f240b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 -- 172.21.15.90:0/3634869221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f241559f0 msgr2=0x7f2f24141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f241559f0 0x7f2f24141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.792+0000 7f2f29044700 1 -- 172.21.15.90:0/3634869221 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f14009550 con 0x7f2f240c0a20 2023-03-03T06:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.793+0000 7f2f29044700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f240c0a20 0x7f2f24143df0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009130 tx=0x7f2f14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.793+0000 7f2f19ffb700 1 -- 172.21.15.90:0/3634869221 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f14003e30 con 0x7f2f240c0a20 2023-03-03T06:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.793+0000 7f2f19ffb700 1 -- 172.21.15.90:0/3634869221 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1401f070 con 0x7f2f240c0a20 2023-03-03T06:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.793+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f240bf780 con 0x7f2f240c0a20 2023-03-03T06:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.793+0000 7f2f19ffb700 1 -- 172.21.15.90:0/3634869221 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1400b8a0 con 0x7f2f240c0a20 2023-03-03T06:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.793+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2406d570 con 0x7f2f240c0a20 2023-03-03T06:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.794+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f240a0f80 con 0x7f2f240c0a20 2023-03-03T06:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.794+0000 7f2f19ffb700 1 -- 172.21.15.90:0/3634869221 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f1402a020 con 0x7f2f240c0a20 2023-03-03T06:18:16.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.794+0000 7f2f19ffb700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f2f08068320 0x7f2f0806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:16.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.795+0000 7f2f19ffb700 1 -- 172.21.15.90:0/3634869221 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2f14087490 con 0x7f2f240c0a20 2023-03-03T06:18:16.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.795+0000 7f2f28843700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f2f08068320 0x7f2f0806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:16.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.795+0000 7f2f28843700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f2f08068320 0x7f2f0806a7d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2f241493a0 tx=0x7f2f20007a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:16.802 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:16.799+0000 7f2f19ffb700 1 -- 172.21.15.90:0/3634869221 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f14053cf0 con 0x7f2f240c0a20 2023-03-03T06:18:17.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.023+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f240a0f80 con 0x7f2f240c0a20 2023-03-03T06:18:17.025 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.024+0000 7f2f19ffb700 1 -- 172.21.15.90:0/3634869221 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4648 (secure 0 0 0) 0x7f2f14057310 con 0x7f2f240c0a20 2023-03-03T06:18:17.026 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-03-03T06:18:17.026 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4250,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/1770068404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":1770068404},{"type":"v1","addr":"172.21.15.121:6834","nonce":1770068404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/1411382210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":1411382210},{"type":"v1","addr":"172.21.15.90:6837","nonce":1411382210}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4290,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/2684520267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":2684520267},{"type":"v1","addr":"172.21.15.121:6835","nonce":2684520267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T06:18:05.752643+0000","modified":"2023-03-03T06:18:08.761036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6836/3337533052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3337533052},{"type":"v1","addr":"172.21.15.90:6836","nonce":3337533052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T06:18:17.027 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f2f08068320 msgr2=0x7f2f0806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f2f08068320 0x7f2f0806a7d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2f241493a0 tx=0x7f2f20007a20 comp rx=0 tx=0).stop 2023-03-03T06:18:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f240c0a20 msgr2=0x7f2f24143df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f240c0a20 0x7f2f24143df0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009130 tx=0x7f2f14004750 comp rx=0 tx=0).stop 2023-03-03T06:18:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 shutdown_connections 2023-03-03T06:18:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f241559f0 0x7f2f24141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f2f08068320 0x7f2f0806a7d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.029 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f240c0a20 0x7f2f24143df0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.029 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 --2- 172.21.15.90:0/3634869221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f24151230 0x7f2f240b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.029 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 >> 172.21.15.90:0/3634869221 conn(0x7f2f240c1900 msgr2=0x7f2f240bd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.029 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 shutdown_connections 2023-03-03T06:18:17.029 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.026+0000 7f2f2b2a8700 1 -- 172.21.15.90:0/3634869221 wait complete. 2023-03-03T06:18:17.030 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2023-03-03T06:18:17.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T06:18:05.752643+0000', 'modified': '2023-03-03T06:18:08.761036+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4297}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4297': {'gid': 4297, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.90:6836/3337533052', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 3337533052}, {'type': 'v1', 'addr': '172.21.15.90:6836', 'nonce': 3337533052}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T06:18:17.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T06:18:17.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-03-03T06:18:17.566 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.564+0000 7fec88386700 1 Processor -- start 2023-03-03T06:18:17.566 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.564+0000 7fec88386700 1 -- start start 2023-03-03T06:18:17.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec88386700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec80072120 0x7fec8006a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec88386700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fec8006a870 0x7fec8006acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec88386700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800662d0 con 0x7fec800c2270 2023-03-03T06:18:17.568 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec88386700 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec800bcf00 con 0x7fec80072120 2023-03-03T06:18:17.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec88386700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec800be0a0 con 0x7fec8006a870 2023-03-03T06:18:17.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec85921700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec80072120 0x7fec8006a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec85921700 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec80072120 0x7fec8006a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34514/0 (socket says 172.21.15.90:34514) 2023-03-03T06:18:17.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.565+0000 7fec85921700 1 -- 172.21.15.90:0/1950510654 learned_addr learned my addr 172.21.15.90:0/1950510654 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:17.569 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec86923700 1 --2- 172.21.15.90:0/1950510654 >> v2:172.21.15.121:3300/0 conn(0x7fec8006a870 0x7fec8006acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85921700 1 -- 172.21.15.90:0/1950510654 >> v2:172.21.15.121:3300/0 conn(0x7fec8006a870 msgr2=0x7fec8006acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85921700 1 --2- 172.21.15.90:0/1950510654 >> v2:172.21.15.121:3300/0 conn(0x7fec8006a870 0x7fec8006acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85921700 1 -- 172.21.15.90:0/1950510654 >> v1:172.21.15.90:6789/0 conn(0x7fec800c2270 legacy=0x7fec800c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85120700 1 -- 172.21.15.90:0/1950510654 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757638383 0 0) 0x7fec800662d0 con 0x7fec800c2270 2023-03-03T06:18:17.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85921700 1 -- 172.21.15.90:0/1950510654 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80072890 con 0x7fec80072120 2023-03-03T06:18:17.570 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85921700 1 --2- 172.21.15.90:0/1950510654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec80072120 0x7fec8006a330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec800be310 tx=0x7fec7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c4353553f52aa6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85120700 1 -- 172.21.15.90:0/1950510654 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c005e30 con 0x7fec80072120 2023-03-03T06:18:17.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85120700 1 -- 172.21.15.90:0/1950510654 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c004030 con 0x7fec80072120 2023-03-03T06:18:17.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec85120700 1 -- 172.21.15.90:0/1950510654 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0043e0 con 0x7fec80072120 2023-03-03T06:18:17.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec88386700 1 -- 172.21.15.90:0/1950510654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec80072120 msgr2=0x7fec8006a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.571 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.566+0000 7fec88386700 1 --2- 172.21.15.90:0/1950510654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec80072120 0x7fec8006a330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec800be310 tx=0x7fec7c009bd0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.567+0000 7fec88386700 1 -- 172.21.15.90:0/1950510654 shutdown_connections 2023-03-03T06:18:17.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.567+0000 7fec88386700 1 --2- 172.21.15.90:0/1950510654 >> v2:172.21.15.121:3300/0 conn(0x7fec8006a870 0x7fec8006acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.567+0000 7fec88386700 1 --2- 172.21.15.90:0/1950510654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec80072120 0x7fec8006a330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.567+0000 7fec88386700 1 -- 172.21.15.90:0/1950510654 >> 172.21.15.90:0/1950510654 conn(0x7fec800a4180 msgr2=0x7fec800c18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.567+0000 7fec88386700 1 -- 172.21.15.90:0/1950510654 shutdown_connections 2023-03-03T06:18:17.572 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.567+0000 7fec88386700 1 -- 172.21.15.90:0/1950510654 wait complete. 2023-03-03T06:18:17.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 Processor -- start 2023-03-03T06:18:17.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 -- start start 2023-03-03T06:18:17.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8006a870 0x7fec801d2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec80072120 0x7fec800a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800c2270 0x7fec800a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.576 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec801d6160 con 0x7fec8006a870 2023-03-03T06:18:17.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec801e3ad0 con 0x7fec800c2270 2023-03-03T06:18:17.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec88386700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec801d6370 con 0x7fec80072120 2023-03-03T06:18:17.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec86122700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8006a870 0x7fec801d2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec86122700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8006a870 0x7fec801d2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34518/0 (socket says 172.21.15.90:34518) 2023-03-03T06:18:17.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.568+0000 7fec86122700 1 -- 172.21.15.90:0/3922239609 learned_addr learned my addr 172.21.15.90:0/3922239609 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:17.577 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86923700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800c2270 0x7fec800a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec85921700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec80072120 0x7fec800a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86122700 1 -- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec80072120 msgr2=0x7fec800a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86122700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec80072120 0x7fec800a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86122700 1 -- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800c2270 msgr2=0x7fec800a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86122700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800c2270 0x7fec800a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.578 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86122700 1 -- 172.21.15.90:0/3922239609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009500 con 0x7fec8006a870 2023-03-03T06:18:17.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86923700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800c2270 0x7fec800a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T06:18:17.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec86122700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8006a870 0x7fec801d2530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec700011e0 tx=0x7fec7000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec76ffd700 1 -- 172.21.15.90:0/3922239609 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec700054e0 con 0x7fec8006a870 2023-03-03T06:18:17.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec76ffd700 1 -- 172.21.15.90:0/3922239609 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70003680 con 0x7fec8006a870 2023-03-03T06:18:17.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8014c620 con 0x7fec8006a870 2023-03-03T06:18:17.579 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec76ffd700 1 -- 172.21.15.90:0/3922239609 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70003a30 con 0x7fec8006a870 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.569+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8014c340 con 0x7fec8006a870 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.570+0000 7fec76ffd700 1 -- 172.21.15.90:0/3922239609 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fec70019030 con 0x7fec8006a870 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.571+0000 7fec76ffd700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fec640682f0 0x7fec6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.571+0000 7fec76ffd700 1 -- 172.21.15.90:0/3922239609 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fec70085070 con 0x7fec8006a870 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.571+0000 7fec85921700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fec640682f0 0x7fec6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.571+0000 7fec85921700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fec640682f0 0x7fec6406a7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009130 tx=0x7fec7c008070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 Processor -- start 2023-03-03T06:18:17.580 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 -- start start 2023-03-03T06:18:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c246ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24c410 0x56247c24a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c266900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c135bc0 con 0x56247c246ad0 2023-03-03T06:18:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c1a4350 con 0x56247c24c410 2023-03-03T06:18:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.573+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x56247c19d880 con 0x56247c24a570 2023-03-03T06:18:17.581 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c246ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c246ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34528/0 (socket says 172.21.15.90:34528) 2023-03-03T06:18:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 -- 172.21.15.90:0/3299687573 learned_addr learned my addr 172.21.15.90:0/3299687573 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec85921700 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24c410 0x56247c24a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86923700 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c266900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 -- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 msgr2=0x56247c266900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c266900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 -- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24c410 msgr2=0x56247c24a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24c410 0x56247c24a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 -- 172.21.15.90:0/3299687573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56247c01f5e0 con 0x56247c246ad0 2023-03-03T06:18:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec86122700 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c246ea0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fec700554c0 tx=0x7fec70054fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a375430444ff8b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec63fff700 1 -- 172.21.15.90:0/3299687573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70053790 con 0x56247c246ad0 2023-03-03T06:18:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec63fff700 1 -- 172.21.15.90:0/3299687573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec700590b0 con 0x56247c246ad0 2023-03-03T06:18:17.583 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.574+0000 7fec63fff700 1 -- 172.21.15.90:0/3299687573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7005d190 con 0x56247c246ad0 2023-03-03T06:18:17.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 -- 172.21.15.90:0/3299687573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 msgr2=0x56247c246ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c246ea0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fec700554c0 tx=0x7fec70054fc0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 -- 172.21.15.90:0/3299687573 shutdown_connections 2023-03-03T06:18:17.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c266900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.587 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c246ea0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3299687573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24c410 0x56247c24a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 -- 172.21.15.90:0/3299687573 >> 172.21.15.90:0/3299687573 conn(0x56247c240720 msgr2=0x56247c2495b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 -- 172.21.15.90:0/3299687573 shutdown_connections 2023-03-03T06:18:17.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 -- 172.21.15.90:0/3299687573 wait complete. 2023-03-03T06:18:17.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 -- start start 2023-03-03T06:18:17.588 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.575+0000 7fec9450fb80 1 Processor -- start 2023-03-03T06:18:17.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c281930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c27e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24c410 0x56247c292530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec9450fb80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c0218d0 con 0x56247c24c410 2023-03-03T06:18:17.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c296500 con 0x56247c246ad0 2023-03-03T06:18:17.589 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x56247c292fb0 con 0x56247c24a570 2023-03-03T06:18:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec9450fb80 10 client.0 _ll_register_callbacks cb 0x56247c13b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T06:18:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec86923700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24c410 0x56247c292530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec86923700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24c410 0x56247c292530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34536/0 (socket says 172.21.15.90:34536) 2023-03-03T06:18:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.576+0000 7fec86923700 1 -- 172.21.15.90:0/3066507175 learned_addr learned my addr 172.21.15.90:0/3066507175 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec85921700 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c27e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86923700 1 -- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 msgr2=0x56247c27e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.590 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86122700 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c281930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86923700 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c27e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86923700 1 -- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 msgr2=0x56247c281930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86923700 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c281930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86923700 1 -- 172.21.15.90:0/3066507175 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8014c340 con 0x56247c24c410 2023-03-03T06:18:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86923700 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24c410 0x56247c292530 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fec78007a90 tx=0x7fec78007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f5869270d93b72 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec617fa700 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2023-03-03T06:18:17.591 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec617fa700 1 -- 172.21.15.90:0/3066507175 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7800e040 con 0x56247c24c410 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec617fa700 1 -- 172.21.15.90:0/3066507175 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fec78012b80 con 0x56247c24c410 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56247c16b140 con 0x56247c24c410 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec617fa700 1 -- 172.21.15.90:0/3066507175 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec78012d40 con 0x56247c24c410 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 10 client.4599 fetch_fsmap learned FSMap version 8 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x56247bfcbd90 con 0x56247c24c410 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec617fa700 1 -- 172.21.15.90:0/3066507175 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56247c16b140 con 0x56247c24c410 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec617fa700 1 -- 172.21.15.90:0/3066507175 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 8) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fec78014040 con 0x56247c24c410 2023-03-03T06:18:17.592 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 10 client.4599 fetch_fsmap finished waiting for FSMap version 8 2023-03-03T06:18:17.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 10 client.4599 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T06:18:17.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 20 client.4599 populate_metadata read hostname 'smithi090' 2023-03-03T06:18:17.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2826b0 0x56247c282a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 4 client.4599 mds_command: new command op to 4297 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T06:18:17.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 --> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56247c021f10 con 0x56247c2826b0 2023-03-03T06:18:17.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.577+0000 7fec86122700 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2826b0 0x56247c282a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.593 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec617fa700 10 client.4599 ms_handle_connect on v2:172.21.15.90:6834/3337533052 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec86122700 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2826b0 0x56247c282a80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=580e01dc9f01c028 server_cookie=17a2f958bab4f8a0 in_seq=0 out_seq=0 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec617fa700 1 -- 172.21.15.90:0/3066507175 <== mds.0 v2:172.21.15.90:6834/3337533052 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x56247c021f10 con 0x56247c2826b0 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec617fa700 10 client.4599 handle_command_reply: tid=0 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 client.4599 shutdown 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24c410 msgr2=0x56247c292530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24c410 0x56247c292530 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fec78007a90 tx=0x7fec78007e50 comp rx=0 tx=0).stop 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 shutdown_connections 2023-03-03T06:18:17.594 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2826b0 0x56247c282a80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24c410 0x56247c292530 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c281930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 --2- 172.21.15.90:0/3066507175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24a570 0x56247c27e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec617fa700 0 client.4599 ms_handle_reset on v2:172.21.15.90:6834/3337533052 2023-03-03T06:18:17.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 >> 172.21.15.90:0/3066507175 conn(0x56247c240720 msgr2=0x56247c246ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 shutdown_connections 2023-03-03T06:18:17.595 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.578+0000 7fec9450fb80 1 -- 172.21.15.90:0/3066507175 wait complete. 2023-03-03T06:18:17.596 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.579+0000 7fec9450fb80 20 client.4599 trim_cache size 0 max 16384 2023-03-03T06:18:17.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.596+0000 7fec9450fb80 1 Processor -- start 2023-03-03T06:18:17.598 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.596+0000 7fec9450fb80 1 -- start start 2023-03-03T06:18:17.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c295700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24a570 0x56247c282a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c282fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec9450fb80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c28ce10 con 0x56247c246ad0 2023-03-03T06:18:17.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c24b0e0 con 0x56247c24a570 2023-03-03T06:18:17.600 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x56247c021f10 con 0x56247c24c410 2023-03-03T06:18:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c295700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c295700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34540/0 (socket says 172.21.15.90:34540) 2023-03-03T06:18:17.601 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 -- 172.21.15.90:0/1140407533 learned_addr learned my addr 172.21.15.90:0/1140407533 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86923700 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c282fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 -- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 msgr2=0x56247c282fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c282fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec85921700 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24a570 0x56247c282a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.602 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 -- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24a570 msgr2=0x56247c282a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24a570 0x56247c282a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 -- 172.21.15.90:0/1140407533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56247c164140 con 0x56247c246ad0 2023-03-03T06:18:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec86122700 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c295700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fec700554c0 tx=0x7fec70084000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad07b660f69119b7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec74daf700 1 -- 172.21.15.90:0/1140407533 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70055300 con 0x56247c246ad0 2023-03-03T06:18:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec74daf700 1 -- 172.21.15.90:0/1140407533 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70053930 con 0x56247c246ad0 2023-03-03T06:18:17.603 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.597+0000 7fec74daf700 1 -- 172.21.15.90:0/1140407533 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec700592f0 con 0x56247c246ad0 2023-03-03T06:18:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 -- 172.21.15.90:0/1140407533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 msgr2=0x56247c295700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c295700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fec700554c0 tx=0x7fec70084000 comp rx=0 tx=0).stop 2023-03-03T06:18:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 -- 172.21.15.90:0/1140407533 shutdown_connections 2023-03-03T06:18:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c282fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.604 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c246ad0 0x56247c295700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 --2- 172.21.15.90:0/1140407533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c24a570 0x56247c282a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 -- 172.21.15.90:0/1140407533 >> 172.21.15.90:0/1140407533 conn(0x56247c240720 msgr2=0x56247c2495b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 -- 172.21.15.90:0/1140407533 shutdown_connections 2023-03-03T06:18:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 -- 172.21.15.90:0/1140407533 wait complete. 2023-03-03T06:18:17.605 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.598+0000 7fec9450fb80 1 -- start start 2023-03-03T06:18:17.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 1 Processor -- start 2023-03-03T06:18:17.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c289560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.608 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24a570 0x56247c27f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c2e4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c27b8b0 con 0x56247c24a570 2023-03-03T06:18:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56247c2800f0 con 0x56247c246ad0 2023-03-03T06:18:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x56247c27feb0 con 0x56247c24c410 2023-03-03T06:18:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec9450fb80 10 client.0 _ll_register_callbacks cb 0x56247c13b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T06:18:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.599+0000 7fec85921700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24a570 0x56247c27f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24a570 0x56247c27f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34546/0 (socket says 172.21.15.90:34546) 2023-03-03T06:18:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 -- 172.21.15.90:0/2350312292 learned_addr learned my addr 172.21.15.90:0/2350312292 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec86122700 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c289560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec86923700 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c2e4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 -- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 msgr2=0x56247c2e4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c2e4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 -- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 msgr2=0x56247c289560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T06:18:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c289560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 -- 172.21.15.90:0/2350312292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56247c01f5e0 con 0x56247c24a570 2023-03-03T06:18:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec85921700 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24a570 0x56247c27f650 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006df0 tx=0x7fec7c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f555c5f7d184a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec617fa700 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2023-03-03T06:18:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec617fa700 1 -- 172.21.15.90:0/2350312292 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c032620 con 0x56247c24a570 2023-03-03T06:18:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec617fa700 1 -- 172.21.15.90:0/2350312292 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fec7c011420 con 0x56247c24a570 2023-03-03T06:18:17.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56247c15b160 con 0x56247c24a570 2023-03-03T06:18:17.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec617fa700 1 -- 172.21.15.90:0/2350312292 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec7c0115e0 con 0x56247c24a570 2023-03-03T06:18:17.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec9450fb80 10 client.4605 fetch_fsmap learned FSMap version 8 2023-03-03T06:18:17.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x56247bff2200 con 0x56247c24a570 2023-03-03T06:18:17.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.600+0000 7fec617fa700 1 -- 172.21.15.90:0/2350312292 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56247c15b160 con 0x56247c24a570 2023-03-03T06:18:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec617fa700 1 -- 172.21.15.90:0/2350312292 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 8) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fec7c032780 con 0x56247c24a570 2023-03-03T06:18:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 10 client.4605 fetch_fsmap finished waiting for FSMap version 8 2023-03-03T06:18:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 10 client.4605 resolve_mds: resolved name 'c' to daemon mds.c 2023-03-03T06:18:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 20 client.4605 populate_metadata read hostname 'smithi090' 2023-03-03T06:18:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2e53b0 0x56247c27dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 4 client.4605 mds_command: new command op to 4297 tid=0[{"prefix": "status"}] 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 --> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56247c145170 con 0x56247c2e53b0 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec86122700 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2e53b0 0x56247c27dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec617fa700 10 client.4605 ms_handle_connect on v2:172.21.15.90:6834/3337533052 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec86122700 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2e53b0 0x56247c27dcb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f1b2699229b7f7f server_cookie=8dbadcac96f26c49 in_seq=0 out_seq=0 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec617fa700 1 -- 172.21.15.90:0/2350312292 <== mds.0 v2:172.21.15.90:6834/3337533052 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x56247c145170 con 0x56247c2e53b0 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec617fa700 10 client.4605 handle_command_reply: tid=0 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 1 client.4605 shutdown 2023-03-03T06:18:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24a570 msgr2=0x56247c27f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.601+0000 7fec9450fb80 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24a570 0x56247c27f650 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006df0 tx=0x7fec7c0047c0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 shutdown_connections 2023-03-03T06:18:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:6834/3337533052,v1:172.21.15.90:6836/3337533052] conn(0x56247c2e53b0 0x56247c27dcb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56247c24c410 0x56247c2e4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56247c246ad0 0x56247c289560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 --2- 172.21.15.90:0/2350312292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56247c24a570 0x56247c27f650 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec617fa700 0 client.4605 ms_handle_reset on v2:172.21.15.90:6834/3337533052 2023-03-03T06:18:17.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 >> 172.21.15.90:0/2350312292 conn(0x56247c240720 msgr2=0x56247c2495b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 shutdown_connections 2023-03-03T06:18:17.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 1 -- 172.21.15.90:0/2350312292 wait complete. 2023-03-03T06:18:17.616 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec9450fb80 20 client.4605 trim_cache size 0 max 16384 2023-03-03T06:18:17.616 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-03-03T06:18:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "05bb16be-ef0c-4f48-a742-b70017592324", 2023-03-03T06:18:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2023-03-03T06:18:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4297, 2023-03-03T06:18:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2023-03-03T06:18:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2023-03-03T06:18:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2023-03-03T06:18:17.618 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 11.847169851, 2023-03-03T06:18:17.618 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 8, 2023-03-03T06:18:17.618 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 23, 2023-03-03T06:18:17.618 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 23, 2023-03-03T06:18:17.618 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 17.157639245999999 2023-03-03T06:18:17.618 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-03-03T06:18:17.619 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fec640682f0 msgr2=0x7fec6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fec640682f0 0x7fec6406a7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009130 tx=0x7fec7c008070 comp rx=0 tx=0).stop 2023-03-03T06:18:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8006a870 msgr2=0x7fec801d2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8006a870 0x7fec801d2530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec700011e0 tx=0x7fec7000cae0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 shutdown_connections 2023-03-03T06:18:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec800c2270 0x7fec800a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7fec640682f0 0x7fec6406a7a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8006a870 0x7fec801d2530 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 --2- 172.21.15.90:0/3922239609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec80072120 0x7fec800a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 >> 172.21.15.90:0/3922239609 conn(0x7fec800a4180 msgr2=0x7fec8006da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 shutdown_connections 2023-03-03T06:18:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.602+0000 7fec88386700 1 -- 172.21.15.90:0/3922239609 wait complete. 2023-03-03T06:18:17.623 INFO:teuthology.run_tasks:Running task kclient... 2023-03-03T06:18:17.680 INFO:tasks.kclient:Mounting kernel clients... 2023-03-03T06:18:17.681 INFO:tasks.kclient:config is {'client.0': {}} 2023-03-03T06:18:17.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:17.682 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2023-03-03T06:18:17.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:17.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2023-03-03T06:18:17.767 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2023-03-03T06:18:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T06:18:17.769 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-03-03T06:18:17.769 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T06:18:17.769 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:17.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T06:18:17.928 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.926+0000 7f18a582e700 1 Processor -- start 2023-03-03T06:18:17.928 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.926+0000 7f18a582e700 1 -- start start 2023-03-03T06:18:17.930 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.927+0000 7f18a582e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f18a006ea90 0x7f18a014f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.927+0000 7f18a582e700 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a0153340 con 0x7f18a014f600 2023-03-03T06:18:17.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.927+0000 7f18a582e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a01497f0 con 0x7f18a0149420 2023-03-03T06:18:17.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.927+0000 7f18a582e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a0152750 con 0x7f18a006ea90 2023-03-03T06:18:17.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.927+0000 7f189f7fe700 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f18a014f600 0x7f18a0159a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53764/0 (socket says 172.21.15.90:53764) 2023-03-03T06:18:17.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.927+0000 7f189f7fe700 1 -- 172.21.15.90:0/4132209608 learned_addr learned my addr 172.21.15.90:0/4132209608 (peer_addr_for_me v1:172.21.15.90:0/0) 2023-03-03T06:18:17.931 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.927+0000 7f189e7fc700 1 --2- 172.21.15.90:0/4132209608 >> v2:172.21.15.121:3300/0 conn(0x7f18a006ea90 0x7f18a014f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814636912 0 0) 0x7f18a0153340 con 0x7f18a014f600 2023-03-03T06:18:17.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18880035b0 con 0x7f18a014f600 2023-03-03T06:18:17.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075931503 0 0) 0x7f18a01497f0 con 0x7f18a0149420 2023-03-03T06:18:17.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a0153340 con 0x7f18a0149420 2023-03-03T06:18:17.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3988782681 0 0) 0x7f18880035b0 con 0x7f18a014f600 2023-03-03T06:18:17.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189e7fc700 1 -- 172.21.15.90:0/4132209608 >> v1:172.21.15.121:6789/0 conn(0x7f18a0149420 legacy=0x7f18a00c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.932 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189e7fc700 1 -- 172.21.15.90:0/4132209608 >> v1:172.21.15.90:6789/0 conn(0x7f18a014f600 legacy=0x7f18a0159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.933 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189e7fc700 1 -- 172.21.15.90:0/4132209608 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0152a10 con 0x7f18a006ea90 2023-03-03T06:18:17.933 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189e7fc700 1 --2- 172.21.15.90:0/4132209608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a006ea90 0x7f18a014f0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1890009840 tx=0x7f1890009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=254251e43514016c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18980019f0 con 0x7f18a006ea90 2023-03-03T06:18:17.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1890004030 con 0x7f18a006ea90 2023-03-03T06:18:17.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.928+0000 7f189dffb700 1 -- 172.21.15.90:0/4132209608 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18900043c0 con 0x7f18a006ea90 2023-03-03T06:18:17.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.929+0000 7f18a582e700 1 -- 172.21.15.90:0/4132209608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a006ea90 msgr2=0x7f18a014f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.929+0000 7f18a582e700 1 --2- 172.21.15.90:0/4132209608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a006ea90 0x7f18a014f0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1890009840 tx=0x7f1890009b50 comp rx=0 tx=0).stop 2023-03-03T06:18:17.934 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.929+0000 7f18a582e700 1 -- 172.21.15.90:0/4132209608 shutdown_connections 2023-03-03T06:18:17.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.929+0000 7f18a582e700 1 --2- 172.21.15.90:0/4132209608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a006ea90 0x7f18a014f0c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.929+0000 7f18a582e700 1 -- 172.21.15.90:0/4132209608 >> 172.21.15.90:0/4132209608 conn(0x7f18a006c0d0 msgr2=0x7f18a00c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:17.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.929+0000 7f18a582e700 1 -- 172.21.15.90:0/4132209608 shutdown_connections 2023-03-03T06:18:17.935 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.929+0000 7f18a582e700 1 -- 172.21.15.90:0/4132209608 wait complete. 2023-03-03T06:18:17.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.930+0000 7f18a582e700 1 Processor -- start 2023-03-03T06:18:17.938 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.930+0000 7f18a582e700 1 -- start start 2023-03-03T06:18:17.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f18a582e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18a006ea90 0x7f18a00bf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f18a582e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a0149420 0x7f18a014baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f18a582e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a014f600 0x7f18a014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.939 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f18a582e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a014ac80 con 0x7f18a014f600 2023-03-03T06:18:17.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f18a582e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a014b310 con 0x7f18a0149420 2023-03-03T06:18:17.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f18a582e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00bf5f0 con 0x7f18a006ea90 2023-03-03T06:18:17.940 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f189f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a014f600 0x7f18a014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f189f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a014f600 0x7f18a014bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34564/0 (socket says 172.21.15.90:34564) 2023-03-03T06:18:17.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f189f7fe700 1 -- 172.21.15.90:0/581268318 learned_addr learned my addr 172.21.15.90:0/581268318 (peer_addr_for_me v2:172.21.15.90:0/0) 2023-03-03T06:18:17.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.931+0000 7f189e7fc700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a0149420 0x7f18a014baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189effd700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18a006ea90 0x7f18a00bf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189f7fe700 1 -- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18a006ea90 msgr2=0x7f18a00bf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.941 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189f7fe700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18a006ea90 0x7f18a00bf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189f7fe700 1 -- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a0149420 msgr2=0x7f18a014baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189f7fe700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a0149420 0x7f18a014baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189f7fe700 1 -- 172.21.15.90:0/581268318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18900094a0 con 0x7f18a014f600 2023-03-03T06:18:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189effd700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18a006ea90 0x7f18a00bf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T06:18:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f189f7fe700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a014f600 0x7f18a014bfe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1898001fa0 tx=0x7f189800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f1887fff700 1 -- 172.21.15.90:0/581268318 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189800ad00 con 0x7f18a014f600 2023-03-03T06:18:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f1887fff700 1 -- 172.21.15.90:0/581268318 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189800ae80 con 0x7f18a014f600 2023-03-03T06:18:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a0152fd0 con 0x7f18a014f600 2023-03-03T06:18:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f1887fff700 1 -- 172.21.15.90:0/581268318 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898007310 con 0x7f18a014f600 2023-03-03T06:18:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.932+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a0152cf0 con 0x7f18a014f600 2023-03-03T06:18:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.933+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a006e610 con 0x7f18a014f600 2023-03-03T06:18:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.933+0000 7f1887fff700 1 -- 172.21.15.90:0/581268318 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f189801a030 con 0x7f18a014f600 2023-03-03T06:18:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.934+0000 7f1887fff700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f188c0682f0 0x7f188c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T06:18:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.934+0000 7f1887fff700 1 -- 172.21.15.90:0/581268318 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1898085c40 con 0x7f18a014f600 2023-03-03T06:18:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.934+0000 7f189effd700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f188c0682f0 0x7f188c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T06:18:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.934+0000 7f189effd700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f188c0682f0 0x7f188c06a7a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1894001750 tx=0x7f1894009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T06:18:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:17.938+0000 7f1887fff700 1 -- 172.21.15.90:0/581268318 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1898052410 con 0x7f18a014f600 2023-03-03T06:18:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.163+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f18a004e780 con 0x7f18a014f600 2023-03-03T06:18:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.164+0000 7f1887fff700 1 -- 172.21.15.90:0/581268318 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1898018020 con 0x7f18a014f600 2023-03-03T06:18:18.166 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-03-03T06:18:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f188c0682f0 msgr2=0x7f188c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f188c0682f0 0x7f188c06a7a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1894001750 tx=0x7f1894009040 comp rx=0 tx=0).stop 2023-03-03T06:18:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a014f600 msgr2=0x7f18a014bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T06:18:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a014f600 0x7f18a014bfe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1898001fa0 tx=0x7f189800a340 comp rx=0 tx=0).stop 2023-03-03T06:18:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 shutdown_connections 2023-03-03T06:18:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:6832/96475,v1:172.21.15.90:6833/96475] conn(0x7f188c0682f0 0x7f188c06a7a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a014f600 0x7f18a014bfe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18a006ea90 0x7f18a00bf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 --2- 172.21.15.90:0/581268318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18a0149420 0x7f18a014baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T06:18:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 >> 172.21.15.90:0/581268318 conn(0x7f18a006c0d0 msgr2=0x7f18a00b7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T06:18:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 shutdown_connections 2023-03-03T06:18:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2023-03-03T06:18:18.166+0000 7f18a582e700 1 -- 172.21.15.90:0/581268318 wait complete. 2023-03-03T06:18:18.179 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-03T06:18:18.179 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T06:18:18.180 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2023-03-03T06:18:18.180 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T06:18:18.180 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:18.180 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-03T06:18:18.180 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T06:18:18.181 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T06:18:18.181 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T06:18:18.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.181 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2023-03-03T06:18:18.198 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T06:18:18.198 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T06:18:18.198 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T06:18:18.198 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2023-03-03T06:18:18.198 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2023-03-03T06:18:18.199 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2023-03-03T06:18:18.199 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T06:18:18.199 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2023-03-03T06:18:18.199 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T06:18:18.199 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2023-03-03T06:18:18.199 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T06:18:18.199 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2023-03-03T06:18:18.200 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T06:18:18.200 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2023-03-03T06:18:18.200 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T06:18:18.200 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42490sec preferred_lft 42490sec 2023-03-03T06:18:18.200 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::1858:e0d9:19dd:b262/64 scope link noprefixroute 2023-03-03T06:18:18.200 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2023-03-03T06:18:18.201 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-03T06:18:18.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.201 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2023-03-03T06:18:18.202 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2023-03-03T06:18:18.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2023-03-03T06:18:18.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2023-03-03T06:18:18.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2023-03-03T06:18:18.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-03T06:18:18.203 DEBUG:teuthology.orchestra.run.smithi090:> ') 2023-03-03T06:18:18.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.370 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T06:18:18.398 INFO:teuthology.orchestra.run.smithi090.stdout:1 2023-03-03T06:18:18.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.399 DEBUG:teuthology.orchestra.run.smithi090:> route 2023-03-03T06:18:18.486 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2023-03-03T06:18:18.487 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T06:18:18.487 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T06:18:18.487 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T06:18:18.487 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T06:18:18.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.488 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2023-03-03T06:18:18.488 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2023-03-03T06:18:18.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T06:18:18.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T06:18:18.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T06:18:18.489 DEBUG:teuthology.orchestra.run.smithi090:> ') 2023-03-03T06:18:18.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.737 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2023-03-03T06:18:18.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.755 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2023-03-03T06:18:18.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.811 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2023-03-03T06:18:18.811 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2023-03-03T06:18:18.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T06:18:18.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-03-03T06:18:18.812 DEBUG:teuthology.orchestra.run.smithi090:> ') 2023-03-03T06:18:18.905 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-03-03T06:18:18.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:18.905 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2023-03-03T06:18:18.906 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2023-03-03T06:18:18.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-03-03T06:18:18.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-03T06:18:18.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-03-03T06:18:18.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-03-03T06:18:18.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-03-03T06:18:18.907 DEBUG:teuthology.orchestra.run.smithi090:> ') 2023-03-03T06:18:19.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:19.122 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2023-03-03T06:18:19.122 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2023-03-03T06:18:19.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2023-03-03T06:18:19.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2023-03-03T06:18:19.124 DEBUG:teuthology.orchestra.run.smithi090:> ') 2023-03-03T06:18:19.222 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T06:18:19.223 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:19.242 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T06:18:19.243 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T06:18:19.243 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:19.299 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-03-03T06:18:19.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:19.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T06:18:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T06:18:19.415 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T06:18:19.531 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T06:18:19.532 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T06:18:19.532 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.121:3300,172.21.15.121:3301" 2023-03-03T06:18:19.532 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with new device syntax: 0@05bb16be-ef0c-4f48-a742-b70017592324.cephfs=/ 2023-03-03T06:18:19.533 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.90:3300/172.21.15.121:3300/172.21.15.121:3301" will pass to kernel 2023-03-03T06:18:19.533 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: v2 mount support check returned 2 2023-03-03T06:18:19.533 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: kernel does not support v2 syntax 2023-03-03T06:18:19.533 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T06:18:19.565 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T06:18:19.565 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T06:18:19.565 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.121:3300,172.21.15.121:3301" 2023-03-03T06:18:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with new device syntax: 0@05bb16be-ef0c-4f48-a742-b70017592324.cephfs=/ 2023-03-03T06:18:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.90:3300/172.21.15.121:3300/172.21.15.121:3301" will pass to kernel 2023-03-03T06:18:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: v2 mount support check returned 2 2023-03-03T06:18:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: kernel does not support v2 syntax 2023-03-03T06:18:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T06:18:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:6789,172.21.15.121:6789,172.21.15.121:6790" 2023-03-03T06:18:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:6789,172.21.15.121:6789,172.21.15.121:6790:/ 2023-03-03T06:18:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=05bb16be-ef0c-4f48-a742-b70017592324" will pass to kernel 2023-03-03T06:18:19.568 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-03-03T06:18:19.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:19.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:19.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:19.604 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T06:18:19.684 INFO:teuthology.orchestra.run.smithi090.stdout:client4614 2023-03-03T06:18:19.684 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:18:19.684 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T06:18:19.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:19.743 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T06:18:19.822 INFO:teuthology.orchestra.run.smithi090.stdout:client4614 2023-03-03T06:18:19.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:19.823 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T06:18:19.902 INFO:teuthology.orchestra.run.smithi090.stdout:client4614 2023-03-03T06:18:19.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T06:18:19.903 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/05bb16be-ef0c-4f48-a742-b70017592324.client4614/status') 2023-03-03T06:18:19.989 INFO:teuthology.orchestra.run.smithi090.stdout:instance: client.4614 (3)192.168.0.1:0/1546264336 2023-03-03T06:18:19.990 INFO:teuthology.orchestra.run.smithi090.stdout:blocklisted: false 2023-03-03T06:18:19.990 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records in 2023-03-03T06:18:19.991 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records out 2023-03-03T06:18:19.991 INFO:teuthology.orchestra.run.smithi090.stderr:69 bytes copied, 2.2069e-05 s, 3.1 MB/s 2023-03-03T06:18:19.991 INFO:teuthology.run_tasks:Running task workunit... 2023-03-03T06:18:20.010 INFO:tasks.workunit:Pulling workunits from ref 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T06:18:20.011 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-03-03T06:18:20.011 INFO:tasks.workunit:timeout=6h 2023-03-03T06:18:20.012 INFO:tasks.workunit:cleanup=True 2023-03-03T06:18:20.013 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:20.030 INFO:teuthology.orchestra.run.smithi090.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:20.031 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-03-03T06:18:20.031 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 2eh/46d Inode: 1 Links: 2 2023-03-03T06:18:20.031 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T06:18:20.031 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:cephfs_t:s0 2023-03-03T06:18:20.031 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-03-03T06:18:20.031 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-03-03 06:18:05.755821537 +0000 2023-03-03T06:18:20.031 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-03-03 06:18:19.600279441 +0000 2023-03-03T06:18:20.032 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2023-03-03 06:18:05.755821537 +0000 2023-03-03T06:18:20.032 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-03-03T06:18:20.032 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-03-03T06:18:20.101 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T06:18:20.158 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-03-03T06:18:34.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:18:34.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:04.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:04.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:09.938 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'. 2023-03-03T06:19:09.939 INFO:tasks.workunit.client.0.smithi090.stderr: 2023-03-03T06:19:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-03-03T06:19:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2023-03-03T06:19:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2023-03-03T06:19:09.939 INFO:tasks.workunit.client.0.smithi090.stderr: 2023-03-03T06:19:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr: 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr: 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr: 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2023-03-03T06:19:09.940 INFO:tasks.workunit.client.0.smithi090.stderr: 2023-03-03T06:19:09.941 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-03-03T06:19:09.941 INFO:tasks.workunit.client.0.smithi090.stderr: 2023-03-03T06:19:09.941 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 472aaef4ec3 Merge PR #45224 into wip-vshankar-testing-20230303.023823 2023-03-03T06:19:09.942 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-03-03T06:19:09.967 INFO:tasks.workunit.client.0.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-03-03T06:19:09.969 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-03T06:19:09.969 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-03-03T06:19:10.685 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-03-03T06:19:10.763 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-03-03T06:19:10.801 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-03T06:19:10.802 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-03T06:19:10.803 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-03-03T06:19:10.839 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-03T06:19:10.842 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-03-03T06:19:10.842 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-03-03T06:19:10.899 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-03-03T06:19:10.900 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-03-03T06:19:10.901 DEBUG:teuthology.orchestra.run.smithi090:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-03-03T06:19:10.961 INFO:tasks.workunit.client.0.smithi090.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-03-03T06:19:13.578 INFO:tasks.workunit.client.0.smithi090.stderr:+ mkdir foo 2023-03-03T06:19:13.583 INFO:tasks.workunit.client.0.smithi090.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-03-03T06:19:13.627 INFO:tasks.workunit.client.0.smithi090.stderr:+ mkdir foo/.snap/barsnap 2023-03-03T06:19:13.651 INFO:tasks.workunit.client.0.smithi090.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-03-03T06:19:13.666 INFO:tasks.workunit.client.0.smithi090.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-03-03T06:19:13.861 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'passed: files are identical' 2023-03-03T06:19:13.862 INFO:tasks.workunit.client.0.smithi090.stdout:passed: files are identical 2023-03-03T06:19:13.862 INFO:tasks.workunit.client.0.smithi090.stderr:+ rmdir foo/.snap/barsnap 2023-03-03T06:19:13.878 INFO:tasks.workunit.client.0.smithi090.stdout:OK 2023-03-03T06:19:13.879 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo OK 2023-03-03T06:19:13.879 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-03-03T06:19:13.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-03-03T06:19:13.946 INFO:tasks.ceph.mds.c.smithi090.stderr:2023-03-03T06:19:13.944+0000 7f334bc55700 -1 mds.0.cache.den(0x601 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray1/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x56490c1e6400) (dversion lock w=1 last_client=4614) pv=23 v=22 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x56490c1d3180] 2023-03-03T06:19:13.946 INFO:tasks.ceph.mds.c.smithi090.stderr:2023-03-03T06:19:13.944+0000 7f334bc55700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x601is fatal because it is a system directory for this rank 2023-03-03T06:19:14.414 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T06:19:14.412+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T06:19:14.414 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T06:19:14.413+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2023-03-03T06:19:14.436 INFO:tasks.ceph.mds.c.smithi090.stderr: -13> 2023-03-03T06:19:13.944+0000 7f334bc55700 -1 mds.0.cache.den(0x601 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray1/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x56490c1e6400) (dversion lock w=1 last_client=4614) pv=23 v=22 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x56490c1d3180] 2023-03-03T06:19:14.436 INFO:tasks.ceph.mds.c.smithi090.stderr: -11> 2023-03-03T06:19:13.944+0000 7f334bc55700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x601is fatal because it is a system directory for this rank 2023-03-03T06:19:14.460 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2023-03-03T06:19:35.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:19:35.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T06:19:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:20:05.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:05.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:35.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:20:35.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:21:05.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:21:05.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:21:35.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:21:35.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:05.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:05.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:35.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:22:35.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:23:05.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:23:05.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:23:35.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:23:35.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:05.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:05.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:35.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:24:35.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:05.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:05.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:36.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:25:36.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:06.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:06.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:36.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:26:36.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:27:06.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:27:06.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:27:36.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:27:36.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:06.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:06.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:36.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:28:36.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:29:06.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:29:06.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:29:36.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:29:36.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T06:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:30:06.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:06.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:36.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:30:36.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:06.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:06.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:36.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:31:36.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:07.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:07.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:37.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:32:37.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:07.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:07.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:37.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:33:37.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:07.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:07.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:37.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:34:37.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:07.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:07.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:37.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:35:37.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:07.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:07.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:37.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:36:37.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:07.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:07.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:37.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:37:37.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:07.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:07.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:38.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:38:38.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:39:08.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:39:08.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:39:38.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:39:38.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T06:40:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:40:08.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:08.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:38.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:40:38.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:08.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:08.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:38.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:41:38.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:08.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:08.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:38.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:42:38.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:08.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:08.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:38.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:43:38.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:08.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:08.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:38.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:44:38.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:45:09.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:45:09.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:45:39.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:45:39.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:09.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:09.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:39.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:46:39.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:09.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:09.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:39.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:47:39.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:09.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:09.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:40.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:48:40.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:10.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:10.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:40.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:49:40.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T06:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T06:50:10.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:10.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:40.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:50:40.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:10.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:10.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:40.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:51:40.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:10.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:10.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:40.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:52:40.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:11.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:11.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:41.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:53:41.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:11.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:11.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:41.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:54:41.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:11.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:11.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:41.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:55:41.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:11.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:11.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:41.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:56:41.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:12.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:12.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:42.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:57:42.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:12.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:12.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:42.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:58:42.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:12.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:12.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:42.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T06:59:42.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T07:00:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:00:12.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:12.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:42.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:00:42.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:13.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:13.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:43.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:01:43.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:13.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:13.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:43.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:02:43.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:13.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:13.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:43.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:03:43.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:13.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:13.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:44.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:04:44.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:14.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:14.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:44.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:05:44.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:14.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:14.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:44.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:06:44.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:14.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:14.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:44.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:07:44.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:15.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:15.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:45.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:08:45.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:15.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:15.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:45.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:09:45.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T07:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:10:15.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:15.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:45.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:10:45.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:16.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:16.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:46.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:11:46.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:16.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:16.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:46.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:12:46.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:16.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:16.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:46.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:13:46.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:14:16.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:14:16.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:14:47.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:14:47.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:17.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:17.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:47.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:15:47.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:16:17.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:16:17.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:16:47.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:16:47.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:17.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:17.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:47.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:17:47.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:17.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:17.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:47.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:18:47.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:19:13.928 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-03-03T07:19:13.929 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-03-03T07:19:18.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:19:18.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:19:48.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:19:48.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T07:20:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:20:18.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:18.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:48.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:20:48.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:18.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:18.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:48.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:21:48.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:18.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:18.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:49.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:22:49.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:23:19.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:23:19.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:23:49.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:23:49.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:19.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:19.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:49.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:24:49.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:19.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:19.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:49.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:25:49.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:26:20.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:26:20.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:26:50.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:26:50.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:20.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:20.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:50.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:27:50.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:20.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:20.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:50.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:28:50.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:29:20.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:29:20.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:29:51.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:29:51.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T07:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:30:21.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:21.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:51.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:30:51.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:31:21.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:31:21.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:31:51.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:31:51.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:21.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:21.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:51.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:32:51.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:21.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:21.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:51.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:33:51.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:22.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:22.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:52.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:34:52.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:22.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:22.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:52.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:35:52.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:22.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:22.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:52.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:36:52.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:22.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:22.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:52.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:37:52.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:38:22.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:38:22.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:38:53.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:38:53.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:23.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:23.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:53.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:39:53.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T07:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:40:23.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:23.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:53.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:40:53.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:23.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:23.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:53.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:41:53.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:24.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:24.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:54.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:42:54.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:24.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:24.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:54.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:43:54.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:24.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:24.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:54.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:44:54.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:45:24.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:45:24.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:45:55.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:45:55.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:25.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:25.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:55.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:46:55.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:25.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:25.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:55.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:47:55.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:25.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:25.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:55.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:48:55.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:25.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:25.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:56.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:49:56.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:50:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T07:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T07:50:26.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:50:26.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:50:56.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:50:56.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:26.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:26.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:56.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:51:56.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:26.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:26.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:56.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:52:56.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:27.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:27.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:57.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:53:57.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:27.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:27.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:57.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:54:57.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:27.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:27.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:57.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:55:57.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:27.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:27.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:57.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:56:57.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:27.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:27.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:58.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:57:58.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:28.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:28.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:58.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:58:58.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:28.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:28.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:58.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T07:59:58.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T07:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:00:28.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:28.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:58.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:00:58.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:28.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:28.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:59.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:01:59.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:29.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:29.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:59.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:02:59.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:29.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:29.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:59.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:03:59.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:29.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:04:29.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:00.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:00.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:30.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:05:30.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:00.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:00.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:30.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:06:30.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:00.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:00.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:30.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:07:30.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:00.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:00.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:30.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:08:30.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:01.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:01.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:31.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:09:31.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T08:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:10:01.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:01.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:31.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:10:31.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:01.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:01.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:31.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:11:31.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:01.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:01.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:31.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:12:31.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:02.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:02.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:32.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:13:32.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:02.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:02.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:32.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:14:32.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:02.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:02.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:32.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:15:32.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:02.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:02.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:33.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:16:33.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:03.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:03.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:33.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:17:33.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:03.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:03.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:33.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:18:33.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:03.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:03.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:33.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:19:33.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T08:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:20:03.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:03.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:33.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:20:33.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:04.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:04.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:34.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:21:34.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:04.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:04.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:34.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:22:34.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:04.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:04.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:34.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:23:34.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:04.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:04.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:35.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:24:35.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:05.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:05.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:35.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:25:35.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:05.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:05.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:35.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:26:35.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:05.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:05.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:35.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:27:35.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:05.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:05.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:36.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:28:36.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:06.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:06.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:36.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:29:36.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T08:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:30:06.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:06.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:36.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:30:36.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:06.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:06.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:36.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:31:36.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:06.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:06.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:37.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:32:37.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:07.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:07.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:37.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:33:37.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:07.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:07.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:37.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:34:37.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:07.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:07.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:37.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:35:37.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:07.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:07.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:38.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:36:38.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:08.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:08.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:38.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:37:38.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:08.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:08.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:38.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:38:38.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:08.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:08.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:38.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:39:38.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T08:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:40:09.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:09.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:39.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:40:39.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:09.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:09.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:39.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:41:39.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:09.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:09.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:39.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:39.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:09.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:09.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:40.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:40.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:10.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:10.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:40.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:40.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:10.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:10.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:40.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:40.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:10.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:10.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:40.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:40.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:10.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:10.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:40.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:40.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:11.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:11.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:41.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:41.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:11.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:11.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:41.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:41.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:00.002 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T08:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T08:50:11.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:11.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:41.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:41.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:11.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:11.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:41.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:41.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:12.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:12.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:42.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:42.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:12.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:12.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:42.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:42.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:12.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:12.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:42.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:42.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:12.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:12.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:43.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:43.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:13.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:13.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:43.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:43.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:13.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:13.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:43.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:43.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:13.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:13.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:43.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:43.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:13.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:13.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:43.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:43.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T08:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:00:14.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:14.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:44.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:44.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:14.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:14.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:44.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:44.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:14.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:14.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:44.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:44.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:14.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:14.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:45.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:45.031 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:15.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:15.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:45.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:45.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:15.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:15.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:45.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:45.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:15.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:15.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:45.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:45.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:15.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:15.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:46.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:46.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:16.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:16.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:46.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:46.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:16.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:16.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:46.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:46.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T09:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:10:16.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:16.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:46.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:46.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:16.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:16.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:47.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:47.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:17.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:17.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:47.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:47.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:17.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:17.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:47.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:47.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:17.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:17.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:47.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:47.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:17.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:17.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:48.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:48.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:18.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:18.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:48.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:48.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:18.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:18.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:48.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:48.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:18.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:18.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:48.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:48.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:19.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:19.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:49.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:49.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T09:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:20:19.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:19.308 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:49.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:49.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:19.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:19.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:49.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:49.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:19.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:19.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:49.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:49.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:20.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:20.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:50.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:50.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:20.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:20.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:50.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:50.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:20.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:20.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:50.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:50.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:20.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:20.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:51.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:51.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:21.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:21.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:51.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:51.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:21.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:21.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:51.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:51.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:21.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:21.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:51.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:51.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T09:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:30:22.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:22.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:52.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:52.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:22.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:22.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:52.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:52.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:22.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:22.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:52.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:52.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:22.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:22.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:52.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:52.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:23.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:23.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:53.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:53.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:23.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:23.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:53.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:53.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:23.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:23.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:53.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:53.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:23.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:23.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:53.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:53.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:23.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:23.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:54.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:54.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:24.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:24.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:54.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:54.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T09:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:40:24.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:24.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:54.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:54.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:24.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:24.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:54.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:54.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:25.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:25.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:55.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:55.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:25.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:25.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:55.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:55.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:25.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:25.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:55.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:55.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:25.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:25.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:56.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:56.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:26.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:26.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:56.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:56.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:26.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:26.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:56.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:56.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:26.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:26.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:56.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:56.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:26.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:26.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:57.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:57.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T09:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T09:50:27.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:27.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:57.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:57.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:27.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:27.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:57.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:57.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:27.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:27.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:57.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:57.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:27.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:27.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:57.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:57.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:28.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:28.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:58.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:58.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:28.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:28.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:58.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:58.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:28.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:28.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:58.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:58.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:28.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:28.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:58.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:58.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:28.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:28.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:59.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:59.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:29.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:29.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:59.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:59.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T09:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:00:29.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:29.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:59.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:59.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:29.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:29.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:59.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:59.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:30.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:30.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:00.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:00.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:30.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:30.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:00.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:00.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:30.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:30.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:00.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:00.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:30.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:30.827 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:00.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:00.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:31.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:31.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:01.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:01.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:31.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:31.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:01.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:01.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:31.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:31.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:01.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:01.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:31.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:31.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T10:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:10:01.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:01.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:32.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:32.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:02.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:02.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:32.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:32.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:02.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:02.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:32.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:32.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:02.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:02.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:32.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:32.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:02.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:02.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:33.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:33.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:03.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:03.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:33.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:33.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:03.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:03.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:33.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:33.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:03.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:03.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:33.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:33.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:04.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:04.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:34.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:34.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:04.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:04.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:34.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:34.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T10:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:20:04.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:04.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:34.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:34.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:04.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:04.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:34.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:34.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:05.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:05.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:35.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:35.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:05.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:05.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:35.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:35.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:05.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:05.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:35.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:35.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:05.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:05.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:36.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:36.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:06.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:06.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:36.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:36.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:06.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:06.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:36.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:36.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:06.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:06.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:36.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:36.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:06.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:06.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:37.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:37.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T10:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:30:07.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:07.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:37.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:37.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:07.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:07.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:37.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:37.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:07.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:07.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:37.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:37.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:07.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:07.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:37.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:37.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:08.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:08.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:38.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:38.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:08.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:08.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:38.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:38.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:08.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:08.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:38.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:38.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:08.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:08.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:38.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:38.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:09.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:09.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:39.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:39.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:09.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:09.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:39.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:39.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T10:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:40:09.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:09.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:39.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:39.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:09.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:09.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:39.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:39.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:10.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:10.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:40.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:40.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:10.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:10.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:40.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:40.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:10.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:10.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:40.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:40.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:10.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:10.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:41.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:41.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:11.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:11.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:41.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:41.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:11.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:11.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:41.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:41.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:11.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:11.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:41.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:41.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:11.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:11.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:42.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:42.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T10:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T10:50:12.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:12.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:42.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:42.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:12.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:12.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:42.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:42.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:12.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:12.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:42.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:42.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:12.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:12.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:43.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:43.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:13.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:13.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:43.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:43.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:13.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:13.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:43.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:43.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:13.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:13.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:43.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:43.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:13.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:13.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:44.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:44.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:14.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:14.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:44.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:44.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:14.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:14.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:44.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:44.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T10:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:00:14.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:14.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:44.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:44.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:14.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:14.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:44.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:44.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:15.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:15.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:45.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:45.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:15.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:15.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:45.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:45.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:15.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:15.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:45.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:45.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:15.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:15.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:45.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:45.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:16.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:16.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:46.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:46.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:16.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:16.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:46.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:46.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:16.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:16.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:46.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:46.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:16.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:16.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:46.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:46.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T11:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:10:17.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:17.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:47.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:47.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:17.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:17.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:47.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:47.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:17.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:17.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:47.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:47.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:17.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:17.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:47.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:47.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:17.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:17.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:47.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:47.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:18.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:18.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:48.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:48.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:18.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:18.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:48.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:48.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:18.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:18.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:48.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:48.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:18.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:18.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:48.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:48.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:18.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:18.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:49.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:49.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T11:19:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:20:19.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:19.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:49.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:49.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:19.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:19.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:49.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:49.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:19.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:19.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:49.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:49.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:19.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:19.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:49.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:49.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:19.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:19.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:49.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:49.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:20.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:20.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:50.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:50.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:20.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:20.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:50.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:50.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:20.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:20.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:50.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:50.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:20.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:20.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:50.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:50.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:21.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:21.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:51.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:51.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T11:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:30:21.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:21.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:51.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:51.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:21.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:21.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:51.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:51.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:21.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:21.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:51.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:51.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:22.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:22.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:52.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:52.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:22.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:22.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:52.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:52.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:22.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:22.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:52.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:52.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:22.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:22.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:52.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:52.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:22.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:22.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:53.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:53.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:23.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:23.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:53.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:53.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:23.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:23.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:53.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:53.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T11:39:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:40:23.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:23.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:53.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:53.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:23.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:23.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:54.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:54.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:24.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:24.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:54.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:54.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:24.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:24.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:54.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:54.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:24.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:24.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:54.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:54.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:24.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:24.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:55.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:55.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:25.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:25.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:55.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:55.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:25.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:25.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:55.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:55.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:25.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:25.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:55.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:55.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:25.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:25.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:56.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:56.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T11:49:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T11:50:26.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:26.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:56.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:56.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:26.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:26.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:56.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:56.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:26.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:26.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:56.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:56.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:26.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:26.827 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:56.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:56.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:27.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:27.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:57.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:57.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:27.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:27.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:57.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:57.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:27.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:27.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:57.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:57.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:27.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:27.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:57.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:57.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:28.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:28.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:58.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:58.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:28.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:28.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:58.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:58.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:00.002 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T11:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:00:28.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:28.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:58.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:58.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:28.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:28.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:58.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:58.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:28.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:28.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:59.003 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:59.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:29.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:29.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:59.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:59.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:29.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:29.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:59.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:59.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:29.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:29.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:59.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:59.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:29.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:29.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:59.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:59.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:29.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:29.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:00.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:00.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:30.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:30.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:00.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:00.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:30.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:30.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T12:09:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:10:00.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:00.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:30.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:30.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:00.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:00.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:30.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:30.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:00.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:00.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:30.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:30.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:01.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:01.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:31.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:31.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:01.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:01.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:31.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:31.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:01.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:01.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:31.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:31.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:01.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:01.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:32.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:32.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:02.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:02.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:32.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:32.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:02.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:02.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:32.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:32.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:02.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:02.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:32.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:32.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T12:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:20:02.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:02.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:32.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:32.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:03.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:03.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:33.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:33.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:03.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:03.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:33.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:33.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:03.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:03.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:33.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:33.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:03.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:03.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:33.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:33.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:03.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:03.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:34.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:34.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:04.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:04.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:34.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:34.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:04.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:04.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:34.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:34.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:04.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:04.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:34.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:34.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:05.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:05.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:35.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:35.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T12:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:30:05.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:05.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:35.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:35.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:05.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:05.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:35.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:35.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:05.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:05.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:36.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:36.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:06.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:06.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:36.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:36.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:06.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:06.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:36.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:36.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:06.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:06.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:37.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:37.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:07.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:07.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:37.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:37.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:07.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:07.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:37.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:37.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:07.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:07.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:37.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:37.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:07.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:07.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:37.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:37.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T12:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:40:07.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:07.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:37.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:37.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:07.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:07.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:38.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:38.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:08.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:08.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:38.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:38.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:08.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:08.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:38.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:38.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:08.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:08.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:38.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:38.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:08.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:08.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:38.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:38.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:09.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:09.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:39.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:39.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:09.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:09.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:39.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:39.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:09.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:09.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:39.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:39.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:09.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:09.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:39.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:39.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T12:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T12:50:09.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:09.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:40.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:40.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:10.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:10.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:40.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:40.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:10.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:10.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:40.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:40.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:10.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:10.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:40.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:40.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:10.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:10.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:41.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:41.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:11.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:11.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:41.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:41.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:11.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:11.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:41.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:41.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:11.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:11.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:41.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:41.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:11.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:11.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:42.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:42.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:12.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:12.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:42.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:42.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T12:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:00:12.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:12.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:42.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:42.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:12.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:12.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:42.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:42.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:13.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:13.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:43.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:43.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:13.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:13.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:43.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:43.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:13.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:13.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:43.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:43.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:13.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:13.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:44.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:44.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:14.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:14.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:44.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:44.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:14.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:14.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:44.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:44.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:14.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:14.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:44.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:44.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:14.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:14.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:44.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:44.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T13:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:10:14.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:14.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:45.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:45.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:15.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:15.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:45.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:45.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:15.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:15.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:45.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:45.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:15.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:15.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:45.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:45.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:15.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:15.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:46.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:46.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:16.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:16.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:46.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:46.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:16.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:16.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:46.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:46.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:16.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:16.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:46.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:46.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:16.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:16.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:47.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:47.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:17.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:17.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:47.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:47.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:00.002 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T13:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:20:17.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:17.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:47.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:47.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:17.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:17.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:47.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:47.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:17.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:17.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:48.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:48.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:18.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:18.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:48.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:48.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:18.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:18.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:48.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:48.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:18.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:18.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:48.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:48.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:18.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:18.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:49.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:49.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:19.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:19.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:49.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:49.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:19.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:19.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:49.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:49.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:19.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:19.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:50.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:50.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T13:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:30:20.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:20.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:50.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:50.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:20.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:20.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:50.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:50.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:20.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:20.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:50.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:50.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:21.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:21.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:51.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:51.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:21.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:21.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:51.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:51.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:21.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:21.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:51.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:51.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:21.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:21.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:51.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:51.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:22.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:22.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:52.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:52.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:22.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:22.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:52.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:52.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:22.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:22.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:52.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:52.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T13:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:40:22.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:22.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:53.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:53.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:23.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:23.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:53.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:53.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:23.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:23.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:53.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:53.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:23.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:23.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:53.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:53.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:23.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:23.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:54.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:54.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:24.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:24.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:54.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:54.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:24.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:24.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:54.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:54.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:24.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:24.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:54.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:54.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:24.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:24.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:55.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:55.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:25.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:25.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:55.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:55.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T13:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T13:50:25.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:25.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:55.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:55.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:25.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:25.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:55.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:55.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:25.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:25.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:56.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:56.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:26.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:26.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:56.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:56.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:26.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:26.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:56.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:56.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:26.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:26.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:56.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:56.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:26.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:26.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:56.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:56.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:26.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:26.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:56.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:56.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:26.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:26.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:57.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:57.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:27.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:27.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:57.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:57.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T13:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:00:27.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:27.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:57.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:57.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:27.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:27.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:58.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:58.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:28.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:28.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:58.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:58.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:28.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:28.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:58.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:58.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:28.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:28.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:58.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:58.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:28.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:28.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:58.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:58.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:29.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:29.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:59.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:59.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:29.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:29.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:59.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:59.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:29.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:29.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:59.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:59.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:29.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:29.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:59.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:59.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:00.002 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T14:09:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:10:30.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:30.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:00.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:00.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:30.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:30.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:00.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:00.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:30.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:30.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:00.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:00.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:30.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:30.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:00.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:00.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:30.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:30.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:01.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:01.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:31.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:31.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:01.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:01.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:31.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:31.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:01.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:01.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:31.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:31.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:01.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:01.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:31.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:31.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:02.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:02.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:32.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:32.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T14:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:20:02.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:02.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:32.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:32.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:02.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:02.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:32.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:32.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:02.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:02.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:32.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:32.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:02.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:02.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:33.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:33.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:03.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:03.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:33.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:33.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:03.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:03.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:33.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:33.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:03.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:03.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:33.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:33.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:03.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:03.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:34.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:34.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:04.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:04.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:34.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:34.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:04.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:04.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:34.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:34.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T14:30:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:30:04.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:04.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:34.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:34.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:05.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:05.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:35.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:35.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:05.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:05.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:35.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:35.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:05.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:05.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:35.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:35.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:05.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:05.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:35.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:35.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:06.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:06.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:36.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:36.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:06.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:06.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:36.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:36.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:06.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:06.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:36.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:36.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:06.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:06.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:36.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:36.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:06.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:06.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:37.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:37.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T14:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:40:07.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:07.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:37.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:37.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:07.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:07.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:37.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:37.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:07.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:07.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:37.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:37.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:07.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:07.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:38.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:38.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:08.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:08.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:38.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:38.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:08.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:08.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:38.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:38.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:08.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:08.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:38.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:38.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:08.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:08.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:39.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:39.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:09.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:09.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:39.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:39.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:09.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:09.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:39.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:39.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T14:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T14:50:09.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:09.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:39.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:39.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:09.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:09.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:39.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:39.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:10.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:10.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:40.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:40.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:10.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:10.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:40.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:40.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:10.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:10.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:40.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:40.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:11.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:11.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:41.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:41.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:11.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:11.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:41.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:41.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:11.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:11.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:41.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:41.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:11.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:11.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:41.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:41.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:12.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:12.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:42.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:42.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T14:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:00:12.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:12.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:42.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:42.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:12.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:12.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:42.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:42.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:12.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:12.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:42.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:42.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:12.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:12.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:43.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:43.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:13.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:13.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:43.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:43.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:13.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:13.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:43.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:43.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:13.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:13.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:43.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:43.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:13.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:13.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:43.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:43.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:14.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:14.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:44.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:44.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:14.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:14.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:44.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:44.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T15:10:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:10:14.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:14.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:44.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:44.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:14.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:14.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:44.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:44.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:15.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:15.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:45.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:45.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:15.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:15.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:45.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:45.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:15.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:15.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:45.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:45.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:15.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:15.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:45.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:45.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:16.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:16.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:46.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:46.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:16.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:16.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:46.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:46.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:16.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:16.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:46.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:46.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:16.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:16.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:46.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:46.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T15:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:20:17.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:17.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:47.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:47.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:17.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:17.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:47.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:47.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:17.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:17.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:47.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:47.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:17.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:17.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:47.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:47.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:18.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:18.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:48.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:48.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:18.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:18.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:48.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:48.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:18.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:18.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:48.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:48.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:18.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:18.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:49.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:49.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:19.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:19.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:49.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:49.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:19.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:19.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:49.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:49.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:00.001 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T15:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:30:19.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:19.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:49.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:49.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:19.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:19.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:49.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:49.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:20.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:20.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:50.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:50.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:20.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:20.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:50.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:50.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:20.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:20.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:50.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:50.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:20.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:20.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:51.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:51.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:21.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:21.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:51.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:51.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:21.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:21.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:51.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:51.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:21.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:21.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:51.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:51.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:21.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:21.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:51.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:51.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T15:39:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:40:21.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:21.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:52.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:52.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:22.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:22.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:52.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:52.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:22.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:22.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:52.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:52.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:22.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:22.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:52.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:52.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:22.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:22.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:52.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:52.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:22.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:22.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:53.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:53.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:23.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:23.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:53.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:53.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:23.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:23.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:53.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:53.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:23.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:23.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:53.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:53.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:23.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:23.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:54.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:54.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T15:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T15:50:24.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:24.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:54.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:54.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:24.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:24.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:54.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:54.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:24.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:24.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:54.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:54.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:25.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:25.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:55.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:55.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:25.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:25.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:55.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:55.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:25.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:25.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:55.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:55.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:25.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:25.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:55.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:55.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:26.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:26.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:56.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:56.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:26.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:26.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:56.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:56.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:26.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:26.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:56.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:56.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T16:00:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:00:26.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:26.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:57.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:57.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:27.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:27.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:57.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:57.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:27.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:27.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:57.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:57.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:27.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:27.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:57.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:57.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:27.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:27.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:57.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:57.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:27.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:27.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:57.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:57.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:27.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:27.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:58.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:58.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:28.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:28.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:58.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:58.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:28.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:28.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:58.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:58.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:28.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:28.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:58.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:58.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T16:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:10:29.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:29.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:59.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:59.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:29.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:29.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:59.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:59.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:29.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:29.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:59.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:59.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:29.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:29.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:59.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:59.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:30.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:30.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:00.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:00.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:30.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:30.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:00.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:00.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:30.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:30.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:00.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:00.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:30.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:30.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:00.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:00.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:30.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:30.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:00.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:00.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:30.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:30.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T16:20:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:20:00.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:01.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:31.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:31.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:01.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:01.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:31.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:31.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:01.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:01.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:31.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:31.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:01.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:01.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:31.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:31.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:01.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:01.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:31.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:31.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:01.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:01.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:31.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:31.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:01.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:01.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:31.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:31.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:02.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:02.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:32.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:32.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:02.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:02.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:32.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:32.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:02.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:02.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:32.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:32.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T16:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:30:02.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:02.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:32.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:32.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:03.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:03.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:33.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:33.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:03.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:03.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:33.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:33.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:03.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:03.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:33.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:33.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:03.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:03.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:33.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:33.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:03.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:03.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:33.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:33.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:03.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:03.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:33.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:33.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:03.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:03.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:33.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:33.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:04.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:04.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:34.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:34.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:04.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:04.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:34.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:34.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T16:39:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:40:04.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:04.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:34.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:34.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:04.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:04.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:34.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:34.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:04.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:04.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:34.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:34.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:04.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:04.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:34.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:34.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:04.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:04.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:34.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:34.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:05.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:05.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:35.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:35.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:05.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:05.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:35.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:35.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:05.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:05.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:35.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:35.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:05.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:05.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:35.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:35.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:05.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:05.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:35.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:35.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T16:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T16:50:06.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:06.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:36.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:36.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:06.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:06.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:36.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:36.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:06.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:06.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:36.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:36.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:06.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:06.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:36.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:36.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:07.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:07.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:37.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:37.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:07.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:07.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:37.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:37.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:07.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:07.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:37.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:37.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:07.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:07.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:37.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:37.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:07.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:07.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:37.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:37.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:07.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:07.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:37.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:37.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T16:59:59.998+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:00:08.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:08.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:38.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:38.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:08.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:08.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:38.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:38.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:08.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:08.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:38.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:38.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:08.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:08.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:38.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:38.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:08.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:08.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:38.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:38.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:08.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:08.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:38.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:38.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:09.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:09.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:39.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:39.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:09.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:09.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:39.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:39.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:09.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:09.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:39.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:39.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:09.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:09.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:39.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:39.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T17:09:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:10:09.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:09.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:40.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:40.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:10.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:10.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:40.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:40.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:10.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:10.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:40.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:40.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:10.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:10.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:40.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:40.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:10.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:10.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:40.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:40.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:11.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:11.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:41.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:41.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:11.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:11.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:41.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:41.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:11.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:11.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:41.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:41.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:11.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:11.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:42.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:42.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:12.157 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:12.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:42.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:42.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T17:19:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:20:12.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:12.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:42.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:42.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:12.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:12.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:42.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:42.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:12.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:12.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:42.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:42.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:13.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:13.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:43.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:43.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:13.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:13.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:43.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:43.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:13.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:13.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:43.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:43.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:13.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:13.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:43.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:43.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:14.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:14.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:44.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:44.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:14.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:14.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:44.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:44.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:14.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:14.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:44.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:44.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:00.000 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T17:29:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:30:14.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:14.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:45.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:45.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:15.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:15.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:45.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:45.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:15.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:15.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:45.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:45.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:15.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:15.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:45.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:45.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:15.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:15.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:45.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:45.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:15.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:15.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:45.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:45.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:16.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:16.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:46.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:46.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:16.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:16.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:46.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:46.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:16.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:16.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:46.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:46.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:16.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:16.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:46.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:46.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T17:40:00.000+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:40:16.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:16.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:46.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:46.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:16.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:16.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:46.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:46.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:16.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:16.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:46.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:46.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:17.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:17.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:47.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:47.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:17.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:17.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:47.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:47.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:17.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:17.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:47.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:47.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:17.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:17.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:47.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:47.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:17.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:17.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:47.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:47.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:17.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:17.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:47.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:47.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:17.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:17.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:48.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:48.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:59.998 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T17:49:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T17:50:18.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:18.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:48.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:48.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:18.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:18.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:48.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:48.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:18.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:18.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:48.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:48.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:18.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:18.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:48.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:48.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:18.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:18.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:48.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:48.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:18.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:18.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:49.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:49.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:19.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:19.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:49.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:49.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:19.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:19.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:49.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:49.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:19.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:19.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:49.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:49.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:19.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:19.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:49.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:49.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-03-03T17:59:59.999+0000 7f97228b2700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged 2023-03-03T18:00:19.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:19.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:49.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:49.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:19.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:19.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:49.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:49.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:19.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:19.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:49.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:49.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:20.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:20.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:50.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:50.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:20.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:20.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:50.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:50.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:20.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:20.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:50.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:50.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:20.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:20.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:50.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:50.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:21.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:21.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:51.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:51.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:54.019 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-03-03T18:07:54.020 DEBUG:teuthology.task.console_log:Killing console logger for smithi090 2023-03-03T18:07:54.021 DEBUG:teuthology.task.console_log:Killing console logger for smithi121 2023-03-03T18:07:54.022 DEBUG:teuthology.exit:Finished running handlers